Thanks for the reply. In an effort to stay with your analogy I would like to ask you the following. If you are driving down the freeway, you do not need a windshield to see. If there was not a windshield there you could still see. Motorcyclist does without them all the time. They are however nice to have so your eyeballs do not dry out. My point is this, I am not 100% convinced that testing is necessary. I know it is ‘evil’ to say such things. However, I would like to know if a team of developers put in rules or guidelines could they stop the need for testing? Is it possible to drive a car without a windshield?

Bob

[James’ Reply: It’s not wind protection I was thinking of, dude. It’s the seeing. Try driving without having any idea where you are or what you are going to run into, eh? That’s my point. In general systems terms, a software project is a cybernetic system. Although it’s possible in principle to produce good software without looking at it, it’s not practical (try and see).

Having suggested it, does it make sense not to have independent and dedicated testers? That depends on the risk. How worried are you? It also depends on your testers. Are they any good? For some companies, they don’t bring testers on board until after there’s been a disaster of some kind.]

Bob

[James’ Reply: Different people may test for different reasons. Testing may have a variety of missions. For my projects it’s usually this: testing is the headlights of the project. We want test because we need to know the status of the product as it travels the “highway” of the project and into the wider world.

In the absence of risk to the project, I usually don’t test.]

“TESLA: Less Filing. Tests Great.”

Ah yes. Today it’d probably be some sort of spreadsheet/html mashup.

[James’ Reply: Actually, yes. These days it’s called FIT.] 

Shouldn’t that be “development tool test team” at Apple? BTW, I greatly enjoy reading your blog when I get a chance which, alas, is all too infrequently. Always thought provoking.

Mark Cleveland

“Basically, we killed assembler.”

[James’ Reply: Hi Mark! There was a reorg in ‘89, sometime around the time of the earthquake. After that, I managed the SPAM team (”Special Projects and Methods”) which was basically a test tool team. Kevin McEntee was on that team. I believe he’s Vice President of Engineering at Netflix, now. He was the architect of our greatest triumph, for which you were the main customer: Tesla, the test language compiler. Remember that? I wish I could have taken that with me when I left Apple. They had no idea what to do with it. Of course, today I’d just use Perl.] 

I would dispute your claim that the pattern — thanks for the correction of the name, and the citation — has little to do with testers. In my opinion, the pattern is a consequence of epistemological scotoma that can be experienced by anyone who decides something is “trivial” or “straightforward” — they’re not thinking about the consequences if they are wrong.

Or haven’t you ever met people who call themselves testers who do that? Even Junit testers?

Regarding the other valuable aphorism you mention: rather than quote scripture (since my copy of Jerry’s book is not at hand at the moment), let me ask the question:

What’s the most pragmatic way of wording it? “is always”? Or “should always be regarded as”?

Because, in the world I have lived in, that finger has very rarely been pointed. I believe the world might be a better place if it were.

But given “Putt’s Law” (the whole book, not just the law), it seems that management reality is defined as what you can get away with. I see managers getting away with an awful lot, just as if that latter aphorism is not at all well understood nor effectively held in mind by those in a position to hold them accountable.

I’ll offer the conjecture that you had a hard time programming your test automation to experience surprise, or to recognize anything as surprising. In my experience, machines react in an equally blasé way to the normal and to the shocking.

In your later post, I believe that you might be referring to “The Universal Pattern of Huge Losses”, found in Chapter 10 of Quality Software Management Volume 2: First Order Measurement. I could be wrong in my belief, since that pattern has little to do with testers and much, much more to do with managers. (In the same paragraph, there’s also the aphorism to the effect that a loss of X dollars is always the responsibility of an executive whose fiscal authority exceeds X dollars.)

Of course, giving burned-out + inexperienced + “eager to impress” people maintenance jobs can be a good way to get more examples of Jerry’s “Universal Law of Large Software Errors”.

[Hmmm, I just Goog’d that phrase and came up empty-handed. Am I remembering it wrong, or is it time for me to add a Wikipedia entry? ]

Another thing I see in help-wanted ads is that small software places want to hire one person to do (all non-unit?) testing *and* be their “buildmeister” — that is, run their periodic code merge and compile-build process.

I’m not sure what all the factors are in that decision, but I wonder if one of them isn’t “Hey, ‘make’ is automated, tests ought to be automated; it’s a perfect skill set match”?

[James’ Reply: I don’t know which Weinberg law you’re referring to. If you Skype me, I bet we an figure it out.] 

The biggest problems I’ve seen come from managers confusing the two activities when estimating and tracking projects. For instance, it’s great when testers and developers cooperate in debugging, but when the testers are spending all day helping developers fix bugs, it screws up estimates based on the assumption that they’re spending all day searching for additional bugs to fix (or ignore).

You can see this problem when there’s a box called “testing” at the end of a project plan, when actually the box ought to be labeled “testing, locating, and fixing, then repeating the process an unknown number of times.” At least then people would realize that there’s no such thing as a waterfall project–only waterfall descriptions of how some managers hope a project will work (for the very first time that ever happened).

IOW, if you’re a tester and don’t want to be blamed for every project screw-up, follow James’s advice and don’t confuse testing and debugging. This is really very important.

[James’ Reply: I like to say there is no such thing as a testing phase– it’s really the fixing phase. I know this because they always ship the product when they think it’s good enough, but not necessarily when they think it’s tested enough.

Good point about estimation. Actually this applies to any project team that divides itself into roles, and then expects everyone to work together, which necessarily involves a blurring of roles. It’s mythical man-month stuff.]

The idea was, in part, to deal with problems such as

(1) Semantic overloading of the terms “pass and fail” — “the product definitely shouldn’t do {x}” — is that a “FAIL” that is really a “PASS”?

(2) The fact that in an evolving product with disputations in specification, the test automation framework can contain test cases that _should_ act one way but don’t yet — is it meaningful to speak of “PASS” and FAIL” when everyone has MIP agreement (”mentioned-in-passing” corridor consensus) that the test case is valid but the product is going to not behave right at present?

Anyone care to guess what my results were? And why?

[James’ Reply: I like the phrase “corridor consensus”.] 

I am not sure if Steve believes that “test automation” is replacement for EVEN what testers ONCE did. Even for once, test automation can replace what a human did while executing a test. Can it? I don’t think so. What is automation script replicated is “one possible” threads of “thinking and action” of what human tester did ONCE.
I believe it is not about doing it once or capturing it in part or whole. No automation tool, I know of can capture what a human tester does while executing a test.

Steve Rowe said “Without it, testers are stuck doing the same things over and over and the products fail to get better.”

What types of testers we are talking about? A skilled tester even when repeating tests can each time focus, observe, model, infer, compare, evaluate, question, hypothesize, check various dimensions, inputs, outputs, interactions of the test. You can not push automation in the pretext that there are tasks, which need to be executed repeatedly without any variation and assessment.
I can think of tasks where I think automation would help - say creating a large data set - I would use tool like perlclip. But this is a tester’s explicit choice that other parameters need to be observed.

Shrini