[James’ Reply: The full answer is a bit long for this space. Here’s the quick version: The two test labs that impressed me were Quardev and LogiGear. In both cases the key ingredients were: a cognitively sophisticated model of testing that can be explained and drawn out on a whiteboard; off-the-job training in the form of classroom lecture and practical exercises; on-the-job mentoring; involvement in the testing community; shunning ISTQB and the other silly certification programs; a self-critical attitude about their own expertise. Instead of claiming “we hire the best”, as most labs do, they say “we don’t know if we’re the best, but here’s what we do to work toward that.”]

Really interesting post (and Blog). I have to say I can definately relate to the “cancellation” point. I would love to hear your thoughts on a new company called uTest - they’re attempting to create a “community testing” model which relates loosely to the time management issues you were discussing. Thanks and I look forward to the feedback.

Best,
Jonathan

[James’ Reply: I don’t know much about it. I’ll ask you the same question I ask any test lab. How do you know your testers are good? How do you know they are good enough? Don’t answer to quickly. Only two test labs (other than STLabs, where I worked) have answered these questions in a way that made me say “cool!” Perhaps you will be the third.]

Of course, what you say is true; but I wasn’t just talking about it as the gotcha game (re that, the Ozzie page mentioned by Graham Shevlin is quite salient).

One of the things about golf is that it’s generally clear to fairly-disposed participants when any particular player’s ball has gone into any particular cup. Neither the cup, nor the ball, nor the behavior of gravity, wind or turf change fundamentally during the course of n holes of golf. I think that’s much of James’s point. SW dev as golf is a species of “ludic fallacy”.

What I was trying to point to is the well-known problem that people speak and think loosely and move goalposts. It’s typical for someone to phrase or interpret things to confirm what they wish to believe. Thus, “Testing phase is 100% complete” == “100% of the alloted testing time and effort has elapsed” turns into a tacit “the product has been tested 100%”.

The last _feels_ so much better, but of course it’s existentially virtually impossible for any complex product.

Redefining the product in order to meet deadlines can also happen in several ways. I expect that every testing practitioner who has been around for any length of time will have participated in some bug triage meeting near the ship date where some pretty important things get deferred.

Sometimes these sorts of things constitute a kind of confirmation bias so strong as to deserve another name entirely. It’s not the blame I’m trying to highlight, it’s the sense of shock some folks seem to experience when there’s a mismatch between what they thought they knew and what turns out to be so.

We teach testers, not just managers, to think that way as a matter of course. If the mission is “find as many problems as you can in an hour (or a test cycle, or a test project)”, then if the tester has found as many problems as she could at the end of the alloted time, then she’s done 100%. The trouble is that managers frequently change the game after the fact when they ask accusingly “why didn’t you find that problem?” The answer, of course, is that we weren’t asked to find that problem. We might have found the problem had it not been hidden so expertly; we might have found it had we not been investing and reporting so many other problems; we might have found it had some of those problems not been in the code to start with. The point is not to blame in return, but to underscore the fact that we haven’t yet found a way to schedule discovery of something hitherto unknown.

Graham Shevlin
http://grahamshevlin.com

Imagine that enough information existed (it might not, but imagine it does) to make the following predictions…

Suppose that data suggests that for the class of project under examination, if it slips one day the odds are 90% that it will slip a whole week, and if it slips a whole week, the odds are 90% that it will slip another three weeks (a month of total slip). Suppose that if it slips a month the odds are 90% that it will slip another three months. 0.9 ^ 3 ==> just under 73% odds that if the project slips a day it will slip four months,

Nobody likes those numbers. So I’d expect them to not be predicted.

Even with different starting odds / assumptions, nobody likes to say there’s any chance of a project slipping too much.

This seems to fit in with two other topics:

1) The “technical debt” issues mentioned elsewhere in this blog (since one way to ship sooner is to redefine completion by cutting features and / or incurring technical debt), and

2) The observation made long ago (by you, James, or was it Jerry Weinberg?) that the only participants in some open-ended testing exercises who claimed 100% of testing had been completed in an hour were the managers — by definition, if one hour of testing was what was scheduled, and one hour of testing was done, that constituted 100% ( ).

Do you remember our discussion at STAR East last year on estimation - Non linear nature of software development and testing tasks - power law, one way valve effect?

All software estimation model (attempts) will fail that do not take into account of nature of software, non linearity of software related tasks ( as against tasks like building/testing non software related things) and human content in software.

Most of the existing models have “linearity” and some notion of counting (test cases, use cases, function points, lines of code etc). For success in developing software estimation model, one has to leave these fundamental assumptions. And there is a “SMC – simple medium, complex” model hoax. I have seen people applying this outrageously simplified model of classifying things – test cases, bugs, automation scripts and so on. Typically the question is “you have 1000 test cases to execute per cycle – how much time does it take?” Immediately, an estimation enthusiast says – “apply SMC model, categorize 1000 cases (some times even worse – use a sample of 30-50 cases of the whole lot) as simple, medium complex and complex test cases, estimate the time for each of the category. Do some math – you will get the estimation. The problem here is typically “complex” means “it takes more time to do. My notion of complexity is degree of “unknown-ness”. Higher the degree of unknown-ness, complex is the stuff.

If we stop counting things, factor in non-linearity, we might be closer to a reasonable software estimation model.

Again, we should stop adopting or barrowing from fields like manufacturing - if it takes 1 hrs to produce a bolt, it would take 100 hrs to produce 100 bolts. Unfortunately, very few managers understand, software is different from a manufacturing assembly line.

Shrini