A few more pairs of focused testing eyes are necessary given that developers in today’s pressurized work environment test at a minimum. We see this all the time. A feature would be code complete and delivered to the QA environment after necessary unit testing and still you find at-least 5-10 bugs with varying severity.

The skills required are different in both these disciplines and one cannot ignore the other.

My 2 cents.

Thanks for the reply. In an effort to stay with your analogy I would like to ask you the following. If you are driving down the freeway, you do not need a windshield to see. If there was not a windshield there you could still see. Motorcyclist does without them all the time. They are however nice to have so your eyeballs do not dry out. My point is this, I am not 100% convinced that testing is necessary. I know it is ‘evil’ to say such things. However, I would like to know if a team of developers put in rules or guidelines could they stop the need for testing? Is it possible to drive a car without a windshield?

Bob

[James' Reply: It's not wind protection I was thinking of, dude. It's the seeing. Try driving without having any idea where you are or what you are going to run into, eh? That's my point. In general systems terms, a software project is a cybernetic system. Although it's possible in principle to produce good software without looking at it, it's not practical (try and see).

Having suggested it, does it make sense not to have independent and dedicated testers? That depends on the risk. How worried are you? It also depends on your testers. Are they any good? For some companies, they don't bring testers on board until after there's been a disaster of some kind.]

Bob

[James' Reply: Different people may test for different reasons. Testing may have a variety of missions. For my projects it's usually this: testing is the headlights of the project. We want test because we need to know the status of the product as it travels the "highway" of the project and into the wider world.

In the absence of risk to the project, I usually don't test.]

“TESLA: Less Filing. Tests Great.”

Ah yes. Today it’d probably be some sort of spreadsheet/html mashup.

[James' Reply: Actually, yes. These days it's called FIT.] 

Shouldn’t that be “development tool test team” at Apple? BTW, I greatly enjoy reading your blog when I get a chance which, alas, is all too infrequently. Always thought provoking.

Mark Cleveland

“Basically, we killed assembler.”

[James' Reply: Hi Mark! There was a reorg in '89, sometime around the time of the earthquake. After that, I managed the SPAM team ("Special Projects and Methods") which was basically a test tool team. Kevin McEntee was on that team. I believe he's Vice President of Engineering at Netflix, now. He was the architect of our greatest triumph, for which you were the main customer: Tesla, the test language compiler. Remember that? I wish I could have taken that with me when I left Apple. They had no idea what to do with it. Of course, today I'd just use Perl.] 

I would dispute your claim that the pattern — thanks for the correction of the name, and the citation — has little to do with testers. In my opinion, the pattern is a consequence of epistemological scotoma that can be experienced by anyone who decides something is “trivial” or “straightforward” — they’re not thinking about the consequences if they are wrong.

Or haven’t you ever met people who call themselves testers who do that? Even Junit testers?

Regarding the other valuable aphorism you mention: rather than quote scripture (since my copy of Jerry’s book is not at hand at the moment), let me ask the question:

What’s the most pragmatic way of wording it? “is always”? Or “should always be regarded as”?

Because, in the world I have lived in, that finger has very rarely been pointed. I believe the world might be a better place if it were.

But given “Putt’s Law” (the whole book, not just the law), it seems that management reality is defined as what you can get away with. I see managers getting away with an awful lot, just as if that latter aphorism is not at all well understood nor effectively held in mind by those in a position to hold them accountable.

I’ll offer the conjecture that you had a hard time programming your test automation to experience surprise, or to recognize anything as surprising. In my experience, machines react in an equally blasé way to the normal and to the shocking.

In your later post, I believe that you might be referring to “The Universal Pattern of Huge Losses”, found in Chapter 10 of Quality Software Management Volume 2: First Order Measurement. I could be wrong in my belief, since that pattern has little to do with testers and much, much more to do with managers. (In the same paragraph, there’s also the aphorism to the effect that a loss of X dollars is always the responsibility of an executive whose fiscal authority exceeds X dollars.)

Of course, giving burned-out + inexperienced + “eager to impress” people maintenance jobs can be a good way to get more examples of Jerry’s “Universal Law of Large Software Errors”.

[Hmmm, I just Goog'd that phrase and came up empty-handed. Am I remembering it wrong, or is it time for me to add a Wikipedia entry? ]

Another thing I see in help-wanted ads is that small software places want to hire one person to do (all non-unit?) testing *and* be their “buildmeister” — that is, run their periodic code merge and compile-build process.

I’m not sure what all the factors are in that decision, but I wonder if one of them isn’t “Hey, ‘make’ is automated, tests ought to be automated; it’s a perfect skill set match”?

[James' Reply: I don't know which Weinberg law you're referring to. If you Skype me, I bet we an figure it out.] 

The biggest problems I’ve seen come from managers confusing the two activities when estimating and tracking projects. For instance, it’s great when testers and developers cooperate in debugging, but when the testers are spending all day helping developers fix bugs, it screws up estimates based on the assumption that they’re spending all day searching for additional bugs to fix (or ignore).

You can see this problem when there’s a box called “testing” at the end of a project plan, when actually the box ought to be labeled “testing, locating, and fixing, then repeating the process an unknown number of times.” At least then people would realize that there’s no such thing as a waterfall project–only waterfall descriptions of how some managers hope a project will work (for the very first time that ever happened).

IOW, if you’re a tester and don’t want to be blamed for every project screw-up, follow James’s advice and don’t confuse testing and debugging. This is really very important.

[James' Reply: I like to say there is no such thing as a testing phase-- it's really the fixing phase. I know this because they always ship the product when they think it's good enough, but not necessarily when they think it's tested enough.

Good point about estimation. Actually this applies to any project team that divides itself into roles, and then expects everyone to work together, which necessarily involves a blurring of roles. It's mythical man-month stuff.]

The idea was, in part, to deal with problems such as

(1) Semantic overloading of the terms “pass and fail” — “the product definitely shouldn’t do {x}” — is that a “FAIL” that is really a “PASS”?

(2) The fact that in an evolving product with disputations in specification, the test automation framework can contain test cases that _should_ act one way but don’t yet — is it meaningful to speak of “PASS” and FAIL” when everyone has MIP agreement (“mentioned-in-passing” corridor consensus) that the test case is valid but the product is going to not behave right at present?

Anyone care to guess what my results were? And why?

[James' Reply: I like the phrase "corridor consensus".] 

I am not sure if Steve believes that “test automation” is replacement for EVEN what testers ONCE did. Even for once, test automation can replace what a human did while executing a test. Can it? I don’t think so. What is automation script replicated is “one possible” threads of “thinking and action” of what human tester did ONCE.
I believe it is not about doing it once or capturing it in part or whole. No automation tool, I know of can capture what a human tester does while executing a test.

Steve Rowe said “Without it, testers are stuck doing the same things over and over and the products fail to get better.â€?

What types of testers we are talking about? A skilled tester even when repeating tests can each time focus, observe, model, infer, compare, evaluate, question, hypothesize, check various dimensions, inputs, outputs, interactions of the test. You can not push automation in the pretext that there are tasks, which need to be executed repeatedly without any variation and assessment.
I can think of tasks where I think automation would help – say creating a large data set – I would use tool like perlclip. But this is a tester’s explicit choice that other parameters need to be observed.

Shrini

bq. And thanks for buying one of my books. I really appreciate it.

Heh. Only the first of several.

This is interesting comment – James. Does this mean mixing debugging and testing could a problem but not a huge problem ? You suggest (in the same comment above) that debugging is process of discovering the underlying fault and removing it. So If “mix” testing and “debugging” – will I not be giving “incorrect” feeling or idea that fixing or removing the fault is part of testing ?

A precisely logged bug might be considered “near equivalent” of “removing the bug – but some one has to fix the code – however small it is and that “small” fix can have its own good or bad side effects.

Am I misinterpreting your words?

Shrini

[James' Reply: I'm not terribly worried to discover that a tester participates in debugging, or performs debugging. I don't see that debugging will necessarily harm testing. An example of mixing testing and debugging is the process of copyediting an article, for instance.

I'm not saying that the two activities should be confused. Don't confuse them, okay?  ]

You mentioned how testing and debugging used to go hand-in-hand. It’s interesting to see how Glen Myers challenged programmers to separate the two processes, but now I’m telling testers to get more involved in debugging (http://tejasconsulting.com/papers/bug-isolation-pnsqc-2004.pdf).

[James' Reply: I see a sharp dividing line between debugging and bug isolation. In the latter, we are dealing only with getting a better description of the behavior. We are not trying to fix it, but rather to sufficiently inform our clients about it. Debugging means discovering the underlying fault and then removing it. It is occasionally the case that precisely describing a bug also happens to identify the fault, such as a bug titled "The FILE menu is spelled FLIE".
I don't see any huge problem with mixing debugging and testing, though.]

[James' Reply: As Don Norman argues in Things That Make Us Smart, adding a tool to a process always changes the process. The invention of the assembly line profoundly changed the craft of manufacturing. You are not replacing what the testers did with your automation, Steve, you are creating a different test process with that automation. We disagree because you don't think you are changing it in a material way, whereas I think you probably are changing it in a material way.

I'm not saying that it's a bad thing to do what you're doing. I also use tools in my work; I am not against tools. What I'm worried about is that you seem not to be aware of, or concerned about, the many things that go on in a good testers mind when a good tester is performing those "repetitive tasks". You apparently see testing as the performance of a physical task (didn't you say something in your post about how using the interface is often the hardest part of testing? I read that and almost choked!), whereas I see testing as a process of continual observation and inference. For you, perhaps it's something like a road rally, where the object is to get through checkpoints as fast as possible, whereas for me, it's not about speed, but richness of experience. I worry about losing that when I apply tools. Aren't you worried, too?

Have you heard about "automation bias"? It afflicts pilots who come to place too much trust in their instrumentation, and lose their ability to reason about their aircraft. There are a number of interesting papers about it. Also, there are beginning to be some interesting incidents, especially in Europe, where people get so into their GPS navigation systems that they take outrageous routes to get to where they are going, and even drive into rivers.

In order to use automation safely, it's important to remind ourselves of what is lost as well as what is gained when we add cruise control to testing.]