Michelle Smith: True Test Leadership

I’m delighted to read Michelle Smith’s play-by-play description of how she is coaching new testers. Take a look.

Let me catalog the coolnesses:

1. “The team I work with was previously exposed to Rapid Software Testing. This exposure caused me to wonder what would happen if these new folks were exposed to some of these ideas early on?”

Notice that she uses the word “wonder”? That’s the attitude I hope to foster in people who take my class. It’s an attitude of curiosity and personal responsibility. She doesn’t speak of applying practices as if the people on the team were milling machines waiting to be programmed. She implies that her testers are learners under their own control. Her attitude is one of establishing a productive but not coercive  relationship.

I don’t know if she got this from my class– probably she had it beforehand– but it’s an attitude I share with her.

2. “I went in their shared office and opened up a five minute conversation with them by asking “What is a bug?” and following that with “who are the people that matter?”.”

Michelle mentions “five minute conversations” a few times. And notice how most of her interactions were in the form of puzzles and questions. It speaks of a light touch with coaching. Light touch is good. Especially with novice testers, experience speaks louder than lecture. Introduce an idea then try it, or try something first, then talk about the idea. Either way, I like how she was getting them working.

3. She had them practice explaining themselves, both in writing and by voice.

4. She was concerned more with the cognitive parts of testing than with the artifacts. That’s good because excellent testing artifacts, such as bug reports and test cases, come from the thinking of the testers. Think well and the rest will follow.

5. She has them work on several aspects of testing. Notice how she deals with oracles, tools, mission, the social process and gaining product knowledge.

I bet what Michelle is doing will lead to better, more passionate testers, and more dynamic, flexible testing. Compare this to what we see so often in our industry: testers simply told to sit down and create test cases. Look, even if you think pre-defined test cases make for great testing, I think to be successful with that you have to base it on skilled and knowledgeable testers. Michelle is creating that foundation with her team.

Overall, what I’m most happy with is that Michelle has made Rapid Testing her own thing. This is vital. This is fundamental to the spirit of my teaching. I want to grow colleagues who confidently think for themselves. Hats off to you, Michelle, for doing that and blogging about it.

Finally, Michelle writes, “I have no idea if what I am doing is going to produce any benefits to them, to the team, or to the stakeholders. Time will tell.”

No best practices nonsense, here. No certification mentality. Just healthy skepticism. Thank you, Michelle!

Quality is Dead #2: The Quality Creation Myth

One of the things that makes it hard to talk about quality software is that we first must overcome the dominating myth about quality, which goes like this: The quality of a product is built into it by its development team. They create quality by following disciplined engineering practices to engineer the source code so that it will fulfill the requirements of the user.

This is a myth, not a lie. It’s a simplified story that helps us make sense of our experience. Myths like this can serve a useful purpose, but we  must take care not to believe in them as if they were the great and hoary truth.

Here are some of the limitations of the myth:

  1. Quality is not a thing and it is not built. To think of it as a thing is to commit the “reification fallacy” that my colleague Michael Bolton loves to hate. Instead, quality is a relationship. Excellent quality is a wonderful sort of relationship. Instead of “building” quality, it’s more coherent to say we arrange for it. Of course you are thinking “what’s the difference between arrange and build? A carpenter could be said to arrange wood into the form of a cabinet. So what?” I like the word arrange because it shifts our attention to relationships and because arrangement suggests less permanence. This is important because in technology we are obliged to work with many elements that are subject to imprecision, ambiguity and drift.
  2. A “practice” is not the whole story of how things get done. To say that we accomplish things by following “practices” or “methods” is to use a figure of speech called a synecdoche– the substitution of a part for the whole. What we call practices are the public face of a lot of shadowy behavior that we don’t normally count as part of the way we work. For instance, joking around, or eating a salad at your desk, or choosing which email to read next, and which to ignore. A social researcher examining a project in progress would look carefully at who talks to whom, how they talk and what they talk about. How is status gained or lost? How do people decide what to do next? What are the dominant beliefs about how to behave in the office? How are documents created and marketed around the team? In what ways do people on the team exert or accept control?
  3. Source code is not the product. The product is the experience that the user receives. That experience comes from the source code in conjunction with numerous other components that are outside the control and sometimes even the knowledge of product developers. It also comes from documentation and support. And that experience plays out over time on what is probably a chaotic multi-tasking computing environment.
  4. “Requirements” are not the requirements, and the “users” are not the users. I don’t know what my requirements are for any of the software I have ever used. I mean, I do know some things. But for anything I think I know, I’m aware that someone else may suggest something that is different that might please me better. Or maybe they will show me how something I thought was important is actually harmful. I don’t know my own requirements for certain. Instead, I make good guesses. Everyone tries to do that. People learn, as they see and work with products, more about what they want. Furthermore, what they want actually changes with their experiences. People change. The users you think you are targeting may not be the users you get.
  5. Fulfillment is not forever and everywhere. The state of the world drifts. A requirement fulfilled today may no longer be fulfilled tomorrow, because  of a new patch to the operating system, or because a new competing product has been released.  Another reason we can’t count on a requirement being fulfilled is that can does not mean will. What I see working with one data set on one computer may not work with other data on another computer.

These factors make certain conversations about quality unhelpful. For instance, I’m impatient when someone claims that unit testing or review will guarantee a great product, because unit testing and review do not account for system level effects, or transient data occurring in the field, or long chains of connected transactions, or intermittent failure of third-party components. Unit testing and review focus on source code. But source code is not the product. So they can be useful, but they are still mere heuristic devices. They provide no guarantee.

Once in a while, I come across a yoho who thinks that a logical specification language like “Z” is the great solution. Because then your specification can be “proven correct.” The big problems with that, of course, is that correctness in this case simply means self-consistency. It does not mean that the specification corresponds to the needs of the customer, nor that it corresponds to the product that is ultimately built.

I’m taking an expansive view of products and projects and quality, because I believe my job is to help people get what they want. Some people, mainly those who go on and on about “disciplined engineering processes” and wish to quantify quality, take a narrower view of their job. I think that’s because their overriding wish is that any problems not be “their fault” but rather YOUR fault. As in, “Hey, I followed the formal spec. If you put the wrong things in the formal spec, that’s YOUR problem, stupid.”

My Take on the Quality Story

Let me offer a more nuanced version of the quality story– still a myth, yes– but one more useful to professionals:

A product is a dynamic arrangement, like a garden that is subject to the elements. A high quality product takes skillful tending and weeding over time. Just like real gardeners, we are not all powerful or all knowing as we grow our crop. We review the conditions and the status of our product as we go. We try to anticipate problems, and we react to solve the problems that occur. We try to understand what our art can and cannot do, and we manage the expectations of our customers accordingly. We know that our product is always subject to decay, and that the tastes of our customers vary. We also know that even the most perfect crop can be spoiled later by a bad chef. Quality, to a significant degree, is out of our hands.

After many years of seeing things work and fail (or work and THEN fail), I think of quality as ephemeral. It may be good enough, at times. It may be better than good enough. But it fades; it always fades, like something natural.

Or like sculpture by Andy Goldsworthy.  (Check out this video.)

This is true for all software, but the degree to which it is a problem will vary. Some systems have been built that work well over time. That is the result of excellent thinking and problem solving on the part of the development team. But I would argue it is also the result of favorable conditions in the surrounding environment. Those conditions are subject to change without notice.

James Tam: Customer Service that Works

After Adam White’s test of of Rypple.com, I decided to try it myself. I soon ran into a fairly serious problem. I was able to try the service without registering, but when I tried to register, the system claimed I already was registered. Then when I tried to reset my password, it claimed I was not registered. Seemed like someone’s database tables were in a bunch.

(Oh well, quality is dead…)

But since I was so hard on WebGreeter about their creepy customer service technology. I thought I’d call Rypple’s toll free line and take my chances.

Aside: I hate calling tech support. I hate the stonewalling and being patronized by poorly trained anonymous dunderheads. Even if I were to lose most of my fingers in a data mining accident I could still count on one hand the good experiences I’ve had with tech support.

Anyway, I had a really good experience. After one ring, James Tam answered. That’s right no voicemail menu! I had tapped the man Tam himself!

The first thing I said was “I want to report a bug on your system.” I was expecting a defensive or brusque reply. Instead he asked me to tell him about it, and I did.

Well here’s the thing. He couldn’t solve my problem right away. He tried. I believe he’s working on it right now. But one thing he said struck me: “This looks like a problem on our side.”

Let me savor those words again…

This looks like a problem on our side.

Somebody taking responsibility?? How often do you hear that from a website support guy? Assuming you ever get to talk to one?

Look what happened. On paper, I had a bad quality experience. Yet I feel good about Rypple. Go Rypple.

Yaron Sinai Says Stop Thinking, Stupid Tester

The Factory School is that community of process people who believe testing benefits from eliminating the human element as much as possible. They wish to mechanize testing, and to condition the humans within it to see themselves as machines and emulate machines as much as possible. It’s an idea that has a number of advantages, with the important caveat that it makes good software testing impossible by leaving no room in the process for skill and thinking.

Sometimes when I complain about the Factory School of software testing, people think I’m exaggerating or making it up.

But check out this quote from Yaron Sinai, CEO of Elementools:

“With Test Case, the team doesn’t think,” Sinai said. “They just need to follow the steps. And for you, as a testing manager, you know that once they completed a set of tests, you know they followed the steps that needed to be followed.”

At first when I saw this, I thought it was a joke news story. Apparently not. (For the sake of Mr. Sinai, I hope he was misquoted. I will gladly post a retraction if that is the case.)

The man’s tool is called Test Case, which is emblematic right there. It’s focus is on test cases, not good testing. His view of test cases appears to be test procedure steps, and he wants testers to stop thinking, dammit, and just follow steps. Like factory robots. Robots that don’t question or talk back. Nice. Saaafe. Rooooooboootttts.

I haven’t found much information about Mr. Sinai online. He apparently hasn’t written about testing, per se. At least he is consistent: he has contributed no ideas to the testing craft, and now he sells an idea prevention system in the guise of a test management tool.

I want to ask Mr. Sinai whether, as a CEO, he faithfully follows his “CEO cases” each day, written for him by other, smarter CEO’s. Or does he, gasp, think for himself? I bet he would reply that although he is a smart CEO, not all CEO’s are smart enough to make their own decisions, and thus it’s only reasonable that their work should be scripted. When I suggest that a minimum requirement to be a CEO should be the ability to think about business problems and make decisions on their own behalf, I’m sure he will say “that’s just not practical.”  By which he will mean, of course, that HE doesn’t know how to do it.

The Factory School promotes the antithesis of engineering, while often using the word engineering as if it were some corpse impaled on a stick. Their approach to managing an engineering process is to kill it. Indeed, a dead process, like a dead horse, is much easier to manage once you get used to the smell.

And a lot of top managers buy this crap because the demos are simple, they are unaware of alternatives that would actually help, and the perfume on their cravats tends to mask the stench of tool vendors who don’t know anything about the craft.

The IMVU Shuffle

Michael Bolton reported on our quick test of IMVU, whose development team brags about having no human-mediated test process before deploying their software to the field.

Some commentors have pointed out that the bugs we found in our twenty minute review weren’t serious– or couldn’t have been– because the IMVU  developers feel successful in what they have produced, and apparently, there are satisfied users of the service.

Hearing that, I’m reminded of the Silver Bridge, which fell down suddenly, one day, after forty years of standing up. Up to that day, it must have seemed quite reasonable to claim that the bridge was a high quality bridge, because– look!– it’s still standing! But lack of information is not proof of excellence, it turns out. That’s why we test. Testing doesn’t provide all possible information, but it provides some. Good testing will provide lots of useful information.

I don’t know if the IMVU system is good enough. I do know that IMVU has no basis to claim that their “continuous integration” process, with all their “automated test cases” has anything to do with their success. By exactly the same “not dead yet” argument, they could justify not running any test cases at all. I can’t help but mention that the finance industry used the same logic to defend deregulation and a weak enforcement of the existing laws that allowed Ponzi schemes and credit swap orders to cripple the world economy. Oops, there goes a few trillion dollars– hey maybe we should have been doing better oversight all these years!

It may be that no possible problem that could be revealed by competent testing would be considered a bad problem byIMVU. If that is the case, then the true reason they are successful is that they have chosen to offer a product that doesn’t matter to people who will accept anything they are offered. Of course, they could use ANY set of practices to do that.

Clearly, what they think they’ve done is establish a test process through automation that will probably discover any important problem that could happen before they release. That’s why Michael and I tested it, and we quickly verified what we expected to find: several problems that materially interfered with the claimed functionality of IMVU, and numerous glitches that suggested the presence of more serious problems nearby. Maybe its present users are willing to put up with it, or maybe they are willing to put up with it for now. But that’s not the point.

The point is that IMVU is not doing certain ordinary and obvious things that would reveal problems in their product and they promote that approach to doing business as if it’s an innovation instead of an evasion of responsibility.

The IMVU people can’t know whether there are, in fact, serious problems in their product because they have chosen not to discover them. That they promote this as a good practice (and that manual testing doesn’t scale, which is also bullshit) tells me that they don’t know what testing is for and they don’t know the difference between testing and a set of computerized behaviors called “test cases”.

They are setting themselves up to rediscover what many others have before them– why we test. Their own experiences will be the best teacher. I predict they will have some doozies.

Quality is Dead #1: The Hypothesis

Quality is dead in computing. Been dead a while, but like some tech’d up version of Weekend at Bernie’s, software purveyors are dressing up its corpse to make us believe computers can bring us joy and salvation.

You know it’s dead, too, don’t you? You long ago stopped expecting anything to just work on your desktop, right? Same here. But the rot has really set in. I feel as if my computer is crawling with maggots. And now it feels that way even when I buy a fresh new computer.

My impression is that up to about ten years ago most companies were still trying, in good faith, to put out a good product. But now many of them, especially the biggest ones, have completely given up. One sign of this is the outsourcing trend. Offshore companies, almost universally, are unwilling and unable to provide solid evidence of their expertise. But that doesn’t matter, because the managers offering them the work care for nothing but the hourly rate of the testers. The ability of the testers to test means nothing. In fact, bright inquisitive testers seem to be frowned upon as troublemakers.

This is my Quality is Dead hypothesis: a pleasing level of quality for end users has become too hard to achieve while demand for it has simultaneously evaporated and penalties for not achieving it are weak. The entropy caused by mindboggling change and innovation in computing has reached a point where it is extremely expensive to use traditional development and testing methods to create reasonably good products and get a reasonable return on investment. Meanwhile, user expectations of quality have been beaten out of them. When I say quality is dead, I don’t mean that it’s dying, or that it’s under threat. What I mean is that we have collectively– and rationally– ceased to expect that software normally works well, even under normal conditions. Furthermore, there is very little any one user can do about it.

(This explains how it is possible for Microsoft to release Vista with a straight face.)

I know of a major U.S. company, that recently laid off a group of more than a dozen trained, talented, and committed testers, instead outsourcing that work to a company in India that obviously does not know how to test (judging from documents shown to me). The management of this well-known American company never talked to their testers or test managers about this (according to the test manager involved and the director above him, both of whom spoke with me). Top management can’t know what they are giving up or what they are getting. They simply want to spend less on testing. When testing becomes just a symbolic ritual, any method of testing will work, as long as it looks impressive to ignorant people and doesn’t cost too much. (Exception: sometimes charging a lot for a fake service is a way to make it seem impressive.)

Please don’t get me wrong. Saving money is not a bad thing. But there are ways to spend less on testing without eviscerating the quality of our work. There are smart ways to outsource, too. What I’m talking about is that this management team obviously didn’t care. They think they can get away with it. And they can: because quality is dead.

I’m also not saying that quality is dead because people in charge are bad people. Instead what we have are systemic incentives that led us to this sorry state, much as did the incentives that resulted in favorable conditions for cholera and plague to sweep across Europe, in centuries past, or the conditions that resulted in the Great Fire of London. It took great disasters to make them improve things.

Witness today how easily the financial managers of the world are evading their responsibility for bringing down the world economy. It’s a similar deal with computing. Weak laws pertaining to quality, coupled with mass fatalism that computers are always going to be buggy, and mass acceptance of ritualistic development and testing practices make the world an unsafe place for users.

If we use computers, or deal with people who do, we are required to adapt to failure and frustration. Our tools of “productivity” suck away our time and confidence. We huddle in little groups on the technological terrain, subject to the whims and mercies of the technically elite. This is true even for members of the technically elite– because being good in one technology does not mean you have much facility with the 5,000 other technologies out there. Each of us is a helpless user, in some respect.

Want an illustration? Just look at my desktop:

  • Software installation is mysterious and fragile. Can I look at any given product on my system and determine if it is properly installed and configured? No.
  • Old data and old bits of applications choke my system. I no longer know for sure what can be thrown away, or where it is. I seem to have three temp folders on my system. What is in them? Why is it there?
  • My task manager is littered with mysterious processes. Going through, googling each one, and cleaning them up is a whole project in and of itself.
  • I once used the Autoruns tool to police my startup. Under Vista, this has become a nightmare. Looking at the Autoruns output is a little like walking into that famous warehouse in Indiana Jones. Which of the buzillion processes are really needed at startup?
  • Mysterious pauses, flickers, and glitches are numerous and ephemeral. Investigating them saps too much time and energy.
  • I see a dozen or two “Is it okay to run this process?” dialog boxes each day, but I never really know if it’s okay.  How could I know? I click YES and hope for the best.
  • I click “I Agree” to EULAs that I rarely read. What rights am I giving away? I have no idea. I’m not qualified to understand most of what’s in those contracts, except they generally disclaim responsibility for quality.
  • Peripherals with proprietary drivers and formats don’t play well with each other.
  • Upgrading to a new computer is now a task comparable with uprooting and moving  to a new city.
  • I’m sick of becoming a power user of each new software package. I want to use my time in other ways, so I remain in a state of ongoing confusion.
  • I am at the mercy of confused computers and their servant who work for credit agencies, utility companies and the government.
  • I have to accept that my personal data will probably be stolen from one of the many companies I do business with online.
  • Proliferating online activity now results in far flung and sometimes forgotten pockets of data about me, clinging like Spanish Moss on the limbs of the Web.

Continuous, low grade confusion and irritation, occasionally spiking to impotent rage, is the daily experience of the technically savvy knowledge worker. I shudder to think what it must be like for computerphobes.

Let me give you one of many examples of what I’m talking about.

I love my Tivo. I was a Tivo customer for three years. So why am I using the Dish Network and not Tivo? The Dish Network DVR sucks. I hate you Dish Network DVR developers! I HATE YOU! HAVEN’T YOU EVER SEEN A TIVO??? DO YOU NOT CARE ABOUT USABILITY AND RELIABILITY, OR ARE YOU TOTAL INCOMPETENT IDIOTS???

I want to use a Tivo, but I can’t use it with the Dish Network. I have to use their proprietary system. I don’t want to use the Dish Network either, but DirectTV was so difficult to deal with for customer service that I refuse to be their customer any more. The guy who installed my Dish Network DVR told me that its “much better than Tivo.” The next time I see him, I want to take him by the scruff of his neck and rub his nose on the screen of my Dish Network DVR as it fails once again to record what I told it to record. You know nothing of Tivos you satellite installer guy! Do not ever criticize Tivo again!

Of all the technology I have knowingly used in the last ten years, I would say I’m most happy with the iPod, the Tivo, and the Neatworks receipt scanning system. My Blackberry has been pretty good, too. Most other things suck.

Quality is dead. What do we do about that? I have some ideas. More to come…

This Blog is Like…

…a radio call-in show. It is not an open public forum.

Sometimes I get complaints about how I handle comments. I received a quite thoughtful and persuasive complaint just a few hours ago. There is a teeny-tiny link on the front page of my blog that goes to my policy on comments, but I suspect few people read it.

Look folks, you are participating in my little show, here. I’m the editorial director. If you make comments, I control whether they go on air. I reject about one in fifteen comments (not counting outright spam) because they violate my guidelines. Think about it like a radio show in text form, and my policy should make sense.

As long as it is not abusive and adds something to the conversation, I want you to have your say. But I WILL argue with you. That’s what I do. And I do that by interpolating my replies into each comment. If you think that’s not fun to see your comments festooned with my replies to you, you are free to do it differently on your own damn blog.

Remember, if you want to unpublish a comment, email me and I’m happy to remove it. If you feel I’m being a bad bad man who abuses people, then I urge you to email me privately and we’ll hash it out. I can’t promise to satisfy you, but I promise to reply to you in good faith.

Those of you with whom I have a working relationship or friendship, feel free to use that to exert influence over my editorial behavior. I am susceptible to charm as long as I feel I’m not being asked to tell lies or muzzle my own ideas.

WebGreeter Fails Turing Test

Beware, if you visit WebGreeter.com a disturbing thing will happen. You will be immediately accosted by what appears to be a chatbot, but is apparently a human doing a creepy impression of a chatbot. This cyborg thing will ask you for your contact information. If you give it to them, they may use it right away to call you on the phone.

Live Attractive Smiling Operator! Click Now!

Live Attractive Smiling Operator! Click Now!

Based on their website, and the high pressure salesman who called me without an invitation, they believe in an aggressive approach to sales. The salesman even used the word “aggressive” several times in his pitch.

The first time I encountered this technology was on a website for expert witnesses. I had trouble using the site and clicked on the “Live Operator” icon. A chat window opened, but everything that they said to me appeared to have come from a script, while seeming to ignore the gist of my questions. I was kind of offended, since this cyborg thing insisted it was a live operator (each time using exactly the same perfectly worded sentence to convey that) while continuing to answer a different question than the one that I asked. At one point I challenged it to type something instead of pasting a canned response, to which I got an obviously canned response that it was not allowed to engage in personal conversation.

(What? Isn’t that the point of a live operator? The ability to engage on a personal level, human-to-human, empathically? Do they think I want a live operator so that I can be ignored in person?)

I concluded that it was not human. But when I went to the WebGreeter website to learn about this weird chatbot technology, I found out it really HAD been a human. That human failed the Turing Test, by successfully declining to use human communication skills with me.

Folks, this is what I’m complaining about with scripting, scripted testing, scripted behavior of any kind. My situation required an appropriately trained and reasonably motivated human to listen to my questions and help me solve my problem. Instead, I meet someone’s idea of an efficient technology that creepily removed what’s good about humanity while being able, technically, to claim humanity. The human in that equation could hide behind rules, pretending to help me while rendering no assistance at all.

I don’t mind that they have scripts. I mind when they are followed instead of applied. This distinction is crucial. In applying, the human remains in charge. In following, the human actually lobotomizes himself in an effort to become animated furniture.

When we deal with people, we ought to be able to trust in certain fundamental human qualities. But just as this crazy WebGreeter company has thrown those out the window, so too have many “process” people torn them out of software projects. Or tried to. Instead they want us to perform rituals. Nice repeatable rituals.