However, with some developers, this can encourage an attitude that the testers are there to “clean up” after the developers. Any bugs that make it through to customers are the tester’s fault because they didn’t catch them. It seems to erode the shared responsibility for quality by skewing the responsibility toward testing rather than balancing it with development. I’ve seen this happen a number of times.

Every time I get a build with new functionality, say a new dialog window, and I press the one button on the dialog and it crashes the software, I wonder about this issue. Am I encouraging the developer in a way that diminishes their responsibility for bugs and hurts the overall quality effort?

[James’ Reply: These are common problems, and they matter. But I find that they can be managed easily without denying me what I need to get my own work done.

We want to encourage programmers to have a high standard in their work by the time that it gets out the door. This doesn’t mean we isolate them until they guarantee it’s a clean product. We are working with them to help them make it clean. Of course working with a team means that people will lean on each other. But we also learn from each other. Leaning and learning are not bad things.]

There is certainly a need for judgement in probing the risks without making an overwhelming number of tests. I read your posts to say those are important human tester skills, along with an expectation that we likely won’t define all the most interesting tests up front.

[James’ Reply: Yes, you seem to get it. Thanks. BTW, the best book on requirements I have seen is Exploring Requirements: Quality Before Design, by Gause and Weinberg. It deals with the need to use a variety of methods to get at the real issues.]

[James’ Reply: Thanks, Jim (or is it Stan? James?). You raise an interesting issue that has a strong role in the philosophy of knowledge: How to distinguish, if at all, between the instrument by which we recognize a “truth” and truth itself?

A short answer is: it gets easier when you consider evidence and risk, rather than truth.

When sorting this out, I find it useful to imagine testing a square root function. Let’s say you have square root function for a 32-bit floating point value. The specification for square roots is clear, and you even have a heuristic oracle, in the form of re-multiplying the square root to arrive at the original value (you still have to deal with rounding errors, but there are algorithms for handling that).

Does a “test suite” that consists of a single test of, say, taking the square root of 4, contain exactly the same information as the definition of square root itself? Clearly it doesn’t, unless square roots are just the same as dividing by 2. Will passing that one test be equivalent to satisfying the requirement of performing square roots? Clearly it won’t. Passing that test suite only tells us that the product is capable of performing square roots in some situation. It’s a question of can vs. will. We will know it can (in some situation); we cannot know it will (in every situation that matters).

If you imagine adding tests to that test suite, the evidence of capability that we are collecting speaks more and more to the question of reliability. The trick in testing is to gather enough of the right kind of evidence to make a well grounded leap of inference from capability to reliability. In the square root case, we must run 2 to the 32 square root cases to have done nearly exhaustive testing (not really exhaustive, but pretty close in some respects). We will then have a pretty strong idea of the reliability of that algorithm, at least with respect to direct input data. We might get away with running fewer tests, depending on the risks we care about.

Problems with specification-by-example and subsequent confusion of examples with tests include accidental inclusion of irrelevant details in the spec and accidental omission of important details. I believe examples are powerful and in most cases necessary, but it’s important to remember that an example is an instance, whereas our software exists to serve vast families of instances.

Quality is equivalent to passed tests only when the tests represent the only input, the only sequence of input, the only combinations of states, the only platform, the only user, the only…{add every other factor here} that you or anyone else who matters cares about.

Coming back to your question: An excellent test process can conceivably stand in for a requirements spec. However, we must always recognize that this our testing is focusing on some things and not others. There is never a perfect alignment. The excellent tester strives to understand the real requirements and seek the real risk, while remaining wary of complacency and lax assumptions.]

I agree that there is little use in postponing a product release to test. As long as there is agreement over how to work on or work around unfinished parts.
However, I do wonder what kind of testing you expect to be performed in a project.

[James’ Reply: Guess!]

You say that you do not demand a programmer to do any unit testing. Does this mean that it is sufficient to perform high level functional testing (both black and white box testing).

[James’ Reply: What is sufficient depends on the situation. It may be sufficient to do no testing at all. I don’t prejudge what is needed. I read the situation and solve the problems that I find there.]

Or should code-level testing of e.g. modules, interfaces etc. also be performed, though not by the programmer but by the tester?
The latter would probably demand quite advanced programming skills from a tester.

[James’ Reply: I have experimented with independent unit-level whitebox testing. The experiment convinced me that it would be very hard to do that productively. It isn’t just a programming skills issue, although there is that. It’s also an issue of keeping the tests up to date with the code, learning the code, and staying coordinated with the programmer. In general, I wouldn’t recommend independent unit-level whitebox testing, but perhaps there are contexts where someone has made it work.]

Some of the discussion seems to mix up the kind of testing you do during the development phase and the kind of testing you do during the formal testing phase. Agile development teams may include testing specialists, but part of the idea of agile development is that everyone on the team is just “the team”, and you don’t want to subdivide the work along the lines of professional specializations. Having a testing specialist on the team is good because he/she can infuse the tester’s perspective into the development process. Developers can learn to write more-testable code, and to think a little bit more like testers. Testers, for their part, can pick up some development skills and gain an appreciation for what, how, and why certain things can go wrong, giving them good information for creating more effective tests. But none of that is the same thing as the formal, after-the-fact testing phase.

[James’ Reply: I’m not convinced that what you are describing is an agile project, Dave. You may be describing an “Agile” project, but that isn’t the same thing. Agility (I’m talking about the English word, not the word made up by some programming consultants that is spelled and pronounced the same) may be served by many approaches and ideas. Personally, I find the “no specialists” rule bizarre. It’s just another way of saying that a homogenous workforce is better than a diversified workforce. It denies education and temperament. It denies experience. I urge you to rethink it.

I like the idea of working as a team. “No specialists” doesn’t mean that. My experience discussing this with Agilists is that “no specialists”, in practice, means that only programming skills are valued.]

At our company, the real value-add of the testing group comes during the formal testing phase that precedes a production release. They are equipped to carry out testing at a level the developers are not. For example, they can test the new or changed solution in the context of a shared server environment alongside the other applications that “live” in that environment. A development team usually is not set up to perform that kind of testing. Their development environment doesn’t mirror the production environment completely, and they don’t have other applications installed there. The testing group is also best equipped to perform comprehensive system testing for performance, maximum load, security, and other factors. Testers also have an objective perspective about the application and won’t make any assumptions about what “should” work.

[James’ Reply: I wonder what role testing skill plays in this. You don’t seem to be talking about it.] 

To enable the testing group to do its job, it’s incumbent on development teams to deliver code that will, at least, run. When development teams deliver code that hasn’t even been unit tested, quite often the testing group has to spend all the time allocated in the project schedule to running unit, integration, and functional tests to expose defects that really should not exist by the time they receive the code. They need to spend their time doing the kind of testing the development team just can’t do. Otherwise they’ll run out of time before they get around to the level of testing they’re really supposed to be doing.

[James’ Reply: It’s great when it runs. However, I don’t need running code to do certain aspects of my work. I don’t want you to delay giving me access to the product because you are concerned about running code. I think I have a different view of my job than you do. If we worked together, I would try to open your eyes to the things I can do with a product that don’t involve running it. I don’t expect you to know much about testing. But I would expect you not to presume to tell me, a tester, what I need in order to do my job. It’s for me to tell you that.]

However you want to divvy up the work is fine, as long as everyone is engaged in a constructive way and doesn’t have a confrontational attitude. I think this is far more important than specific procedures or techniques for development or testing. James’ original post states, “I worry that anything that may cause the programmers to avoid working with me is toxic to rapid, excellent testing.” A negative attitude toward others who are working in different roles than yourself certainly qualifies as “toxic.” Insectivorous’ comments illustrate this point very succinctly. He doesn’t sound like a person I would want to work with, whether as a tester or a developer or in any other role. People with that sort of attitude can destroy a project, whether they are working as developers, testers, managers, or whatever.

[James’ Reply: I think Insectivorous was not talking to you, a programmer. He was talking to his fellow testers. The spirit of Insectivorous’ words, I think, has to do with recognizing what a testing specialist brings to the table. You seem like a nice guy, Dave, so I’d like you to consider that your views on testing may come across as patronizing to those of us who like being testers and take our testing skills seriously. I bet that is not your intention.

In case you didn’t know, I also am a programmer. I am not a programmer-philosopher, though, and I don’t write about how programmer’s should do their work. I don’t feel qualified to do that. I’m surprised at how many programmers feel qualified to tell me how to do testing, not having ever studied the subject, and having no experience beyond intuitive plunking. I’m not sure if you are one of those; I hope you’re not.]

The catch with your way of working is that it will only work with highly skilled testers that at the same time know a lot about programming. That leaves some 99 percent of the testers I have come in contact with including myself. Yes, I did some programming a long time ago but do not know enough to throw myself over half-baked products. That would for me to sped my time badly. However I would like to share what works for me. “My process”

Early involvment in the project: I start reading any piece of information that exists so far on the project and start analysing. Sometimes i visualise it by creating some sort of model. Then I start asking questions regarding what is unclear to me. This way a lt of missing or wrong requirements are noticed and updated. Sometimes I read through the program spec. It really depends on how readable it is for me. Formulas and logic I can understand. language specifics I leave alone. I do find some stuff, mostly due to sloppy work.

Next thing is to volonteer to create test cases for each module, program or whatever piece of code there is on a level that I can understand. Few programmers do any “real testing” other than the most obvious things. Then I hand these test cases over to the programmer so they can run them in THEIR own environment when THEY feel ready to do so. This way nobody has to know whatever embarrassing bugs they put in the code, no defect reports are written, no statistics saved. However I know that they have actually run some pretty good tests AND corrected the defects they found. which means that when they hand over their code to the test team we will find less simple bugs and are allowed to concentrate on more hogh level tests. They look good, testers are happy, management are thrilled.

Then I create test cases for higher levels as System and acceptance test. Some are fairly detailed but i tend to write down less information nowadays. Fully detailed test cases can be handed over to customers, testers with low skills and also the developers themselves. test cases with less detail are run be me or other highly skilled testers who does not want or need too much control. I think they are closer to what you call “charters” than they way they are described in the books you dislike. Developers often run some of the system test cases just to ensure that they will continue looking good, not creating broken builds etc. Like a smoke test on each build. How it works? Great! The test team find fewer bugs, but since we know it is because there are less to be found we are still happy, customers find almost no bugs at all in their acceptance test, if they find any they are usually due to bad requirements…

So I too feel the need to cooperate with developers using my superior skills in testing together with their superior skills in programming. Like you say yourself -”testers must add value”. I think that your way of working works for you and a few other “blingual” tester-developers but my way will work for a larger part - but not all - of the current tester community. Or maybe it can be described on a scale like exploratory-scripted, where you are on the far end of the scale and I am closer to the middle and the certification people and other famous authors are on the other end?

/Toby

PS: How did you spend your spare time before you had comments turned on to your blog

[James’ Reply: If all comments were like this, I would have more spare time. Good points.]

[James’ Reply: On behalf of skilled and diligent testers everywhere, please accept my apologies. Indeed, a skill-free tester is not able to do much with a partly-baked product. My heuristic of instant release applies to me, testers who work for me, and testers who are like me, but not to every tester. It is something I’m suggesting that testers say to their programmer counterparts; something I hope we aspire to, in any case. I’d like to see no distinction between the testing phase and the programming phase. The whole concept of separate phases seems to me based on a myopic and defensive idea of what programming and testing can be.]

This topic creates a gray area for delimiting the end of development testing and the start of system testing.

We need to make sure that the acceptance criteria is clearly defined for everyone throughout our development effort so that the developers not only test the technicalities and architectural side of code (conformance to specifications), but also that their code implements the desired functionality for the end users (conformance to user needs and wants).

[James’ Reply: I’m interested in where this need comes from, for you, since I have not experienced a context where this need exists. I find that acceptance criteria is not a pre-requisite for testing, but rather that testing is a process of refining acceptance criteria. By the end of testing, ideally, we know what we are accepting and why we are accepting it.

Also, as I’ve already written, I don’t particularly need the programmers to do any testing. I appreciate it when they do, just as a paramedic appreciates when people on the scene render CPR to a heart attack victim, but don’t delay calling me in because you feel like you should test. Don’t delay. That’s my message to the programmers I work with.]

This knowledge can be leveraged by jointly reviewing all documents directly derived from requirements: use cases, design specifications, functional specifications, technical specifications, test cases, and of course, the requirements specification document itself.

Developers must be completely aware of what is expected of their code, and to achieve this, cross-functional meetings can be organized before and after every phase of the SDLC to: prevent defect injection, identify defect predictability patterns, and minimize heavy reliance on quality control to find bugs.

By doing this, we are empowering the creators of every deliverable in the SDLC with the knowledge to produce the best quality interim products in terms of compliance to standards, and customer satisfaction (and yes, that includes code).

[James’ Reply: In general, I don’t find that quality software comes from extensive planning and documentation. I think that’s because so much of the extensive planning is bad planning, and so much documentation is bad documentation. I suppose, like cocaine, it’s possible to indulge in that stuff and not get in trouble, but mostly I just see people get in trouble with it. I prefer a more agile and incremental approach to achieve excellent quality software.]

We are together in this software development thing; let everybody do what they do best and remember that everybody else can learn something from it.

[James’ Reply: I don’t follow your reasoning on this. But anyway, have you considered the downside of total independence? There’s always a tradeoff, don’t you think?]