However, with some developers, this can encourage an attitude that the testers are there to “clean up” after the developers. Any bugs that make it through to customers are the tester’s fault because they didn’t catch them. It seems to erode the shared responsibility for quality by skewing the responsibility toward testing rather than balancing it with development. I’ve seen this happen a number of times.

Every time I get a build with new functionality, say a new dialog window, and I press the one button on the dialog and it crashes the software, I wonder about this issue. Am I encouraging the developer in a way that diminishes their responsibility for bugs and hurts the overall quality effort?

[James' Reply: These are common problems, and they matter. But I find that they can be managed easily without denying me what I need to get my own work done.

We want to encourage programmers to have a high standard in their work by the time that it gets out the door. This doesn't mean we isolate them until they guarantee it's a clean product. We are working with them to help them make it clean. Of course working with a team means that people will lean on each other. But we also learn from each other. Leaning and learning are not bad things.]

There is certainly a need for judgement in probing the risks without making an overwhelming number of tests. I read your posts to say those are important human tester skills, along with an expectation that we likely won’t define all the most interesting tests up front.

[James' Reply: Yes, you seem to get it. Thanks. BTW, the best book on requirements I have seen is Exploring Requirements: Quality Before Design, by Gause and Weinberg. It deals with the need to use a variety of methods to get at the real issues.]

[James' Reply: Thanks, Jim (or is it Stan? James?). You raise an interesting issue that has a strong role in the philosophy of knowledge: How to distinguish, if at all, between the instrument by which we recognize a "truth" and truth itself?

A short answer is: it gets easier when you consider evidence and risk, rather than truth.

When sorting this out, I find it useful to imagine testing a square root function. Let's say you have square root function for a 32-bit floating point value. The specification for square roots is clear, and you even have a heuristic oracle, in the form of re-multiplying the square root to arrive at the original value (you still have to deal with rounding errors, but there are algorithms for handling that).

Does a "test suite" that consists of a single test of, say, taking the square root of 4, contain exactly the same information as the definition of square root itself? Clearly it doesn't, unless square roots are just the same as dividing by 2. Will passing that one test be equivalent to satisfying the requirement of performing square roots? Clearly it won't. Passing that test suite only tells us that the product is capable of performing square roots in some situation. It's a question of can vs. will. We will know it can (in some situation); we cannot know it will (in every situation that matters).

If you imagine adding tests to that test suite, the evidence of capability that we are collecting speaks more and more to the question of reliability. The trick in testing is to gather enough of the right kind of evidence to make a well grounded leap of inference from capability to reliability. In the square root case, we must run 2 to the 32 square root cases to have done nearly exhaustive testing (not really exhaustive, but pretty close in some respects). We will then have a pretty strong idea of the reliability of that algorithm, at least with respect to direct input data. We might get away with running fewer tests, depending on the risks we care about.

Problems with specification-by-example and subsequent confusion of examples with tests include accidental inclusion of irrelevant details in the spec and accidental omission of important details. I believe examples are powerful and in most cases necessary, but it's important to remember that an example is an instance, whereas our software exists to serve vast families of instances.

Quality is equivalent to passed tests only when the tests represent the only input, the only sequence of input, the only combinations of states, the only platform, the only user, the only...{add every other factor here} that you or anyone else who matters cares about.

Coming back to your question: An excellent test process can conceivably stand in for a requirements spec. However, we must always recognize that this our testing is focusing on some things and not others. There is never a perfect alignment. The excellent tester strives to understand the real requirements and seek the real risk, while remaining wary of complacency and lax assumptions.]

I agree that there is little use in postponing a product release to test. As long as there is agreement over how to work on or work around unfinished parts.
However, I do wonder what kind of testing you expect to be performed in a project.

[James' Reply: Guess!]

You say that you do not demand a programmer to do any unit testing. Does this mean that it is sufficient to perform high level functional testing (both black and white box testing).

[James' Reply: What is sufficient depends on the situation. It may be sufficient to do no testing at all. I don't prejudge what is needed. I read the situation and solve the problems that I find there.]

Or should code-level testing of e.g. modules, interfaces etc. also be performed, though not by the programmer but by the tester?
The latter would probably demand quite advanced programming skills from a tester.

[James' Reply: I have experimented with independent unit-level whitebox testing. The experiment convinced me that it would be very hard to do that productively. It isn't just a programming skills issue, although there is that. It's also an issue of keeping the tests up to date with the code, learning the code, and staying coordinated with the programmer. In general, I wouldn't recommend independent unit-level whitebox testing, but perhaps there are contexts where someone has made it work.]

Some of the discussion seems to mix up the kind of testing you do during the development phase and the kind of testing you do during the formal testing phase. Agile development teams may include testing specialists, but part of the idea of agile development is that everyone on the team is just “the team”, and you don’t want to subdivide the work along the lines of professional specializations. Having a testing specialist on the team is good because he/she can infuse the tester’s perspective into the development process. Developers can learn to write more-testable code, and to think a little bit more like testers. Testers, for their part, can pick up some development skills and gain an appreciation for what, how, and why certain things can go wrong, giving them good information for creating more effective tests. But none of that is the same thing as the formal, after-the-fact testing phase.

[James' Reply: I'm not convinced that what you are describing is an agile project, Dave. You may be describing an "Agile" project, but that isn't the same thing. Agility (I'm talking about the English word, not the word made up by some programming consultants that is spelled and pronounced the same) may be served by many approaches and ideas. Personally, I find the "no specialists" rule bizarre. It's just another way of saying that a homogenous workforce is better than a diversified workforce. It denies education and temperament. It denies experience. I urge you to rethink it.

I like the idea of working as a team. "No specialists" doesn't mean that. My experience discussing this with Agilists is that "no specialists", in practice, means that only programming skills are valued.]

At our company, the real value-add of the testing group comes during the formal testing phase that precedes a production release. They are equipped to carry out testing at a level the developers are not. For example, they can test the new or changed solution in the context of a shared server environment alongside the other applications that “live” in that environment. A development team usually is not set up to perform that kind of testing. Their development environment doesn’t mirror the production environment completely, and they don’t have other applications installed there. The testing group is also best equipped to perform comprehensive system testing for performance, maximum load, security, and other factors. Testers also have an objective perspective about the application and won’t make any assumptions about what “should” work.

[James' Reply: I wonder what role testing skill plays in this. You don't seem to be talking about it.] 

To enable the testing group to do its job, it’s incumbent on development teams to deliver code that will, at least, run. When development teams deliver code that hasn’t even been unit tested, quite often the testing group has to spend all the time allocated in the project schedule to running unit, integration, and functional tests to expose defects that really should not exist by the time they receive the code. They need to spend their time doing the kind of testing the development team just can’t do. Otherwise they’ll run out of time before they get around to the level of testing they’re really supposed to be doing.

[James' Reply: It's great when it runs. However, I don't need running code to do certain aspects of my work. I don't want you to delay giving me access to the product because you are concerned about running code. I think I have a different view of my job than you do. If we worked together, I would try to open your eyes to the things I can do with a product that don't involve running it. I don't expect you to know much about testing. But I would expect you not to presume to tell me, a tester, what I need in order to do my job. It's for me to tell you that.]

However you want to divvy up the work is fine, as long as everyone is engaged in a constructive way and doesn’t have a confrontational attitude. I think this is far more important than specific procedures or techniques for development or testing. James’ original post states, “I worry that anything that may cause the programmers to avoid working with me is toxic to rapid, excellent testing.” A negative attitude toward others who are working in different roles than yourself certainly qualifies as “toxic.” Insectivorous’ comments illustrate this point very succinctly. He doesn’t sound like a person I would want to work with, whether as a tester or a developer or in any other role. People with that sort of attitude can destroy a project, whether they are working as developers, testers, managers, or whatever.

[James' Reply: I think Insectivorous was not talking to you, a programmer. He was talking to his fellow testers. The spirit of Insectivorous' words, I think, has to do with recognizing what a testing specialist brings to the table. You seem like a nice guy, Dave, so I'd like you to consider that your views on testing may come across as patronizing to those of us who like being testers and take our testing skills seriously. I bet that is not your intention.

In case you didn't know, I also am a programmer. I am not a programmer-philosopher, though, and I don't write about how programmer's should do their work. I don't feel qualified to do that. I'm surprised at how many programmers feel qualified to tell me how to do testing, not having ever studied the subject, and having no experience beyond intuitive plunking. I'm not sure if you are one of those; I hope you're not.]

The catch with your way of working is that it will only work with highly skilled testers that at the same time know a lot about programming. That leaves some 99 percent of the testers I have come in contact with including myself. Yes, I did some programming a long time ago but do not know enough to throw myself over half-baked products. That would for me to sped my time badly. However I would like to share what works for me. “My process”

Early involvment in the project: I start reading any piece of information that exists so far on the project and start analysing. Sometimes i visualise it by creating some sort of model. Then I start asking questions regarding what is unclear to me. This way a lt of missing or wrong requirements are noticed and updated. Sometimes I read through the program spec. It really depends on how readable it is for me. Formulas and logic I can understand. language specifics I leave alone. I do find some stuff, mostly due to sloppy work.

Next thing is to volonteer to create test cases for each module, program or whatever piece of code there is on a level that I can understand. Few programmers do any “real testing” other than the most obvious things. Then I hand these test cases over to the programmer so they can run them in THEIR own environment when THEY feel ready to do so. This way nobody has to know whatever embarrassing bugs they put in the code, no defect reports are written, no statistics saved. However I know that they have actually run some pretty good tests AND corrected the defects they found. which means that when they hand over their code to the test team we will find less simple bugs and are allowed to concentrate on more hogh level tests. They look good, testers are happy, management are thrilled.

Then I create test cases for higher levels as System and acceptance test. Some are fairly detailed but i tend to write down less information nowadays. Fully detailed test cases can be handed over to customers, testers with low skills and also the developers themselves. test cases with less detail are run be me or other highly skilled testers who does not want or need too much control. I think they are closer to what you call “charters” than they way they are described in the books you dislike. Developers often run some of the system test cases just to ensure that they will continue looking good, not creating broken builds etc. Like a smoke test on each build. How it works? Great! The test team find fewer bugs, but since we know it is because there are less to be found we are still happy, customers find almost no bugs at all in their acceptance test, if they find any they are usually due to bad requirements…

So I too feel the need to cooperate with developers using my superior skills in testing together with their superior skills in programming. Like you say yourself -”testers must add value”. I think that your way of working works for you and a few other “blingual” tester-developers but my way will work for a larger part – but not all – of the current tester community. Or maybe it can be described on a scale like exploratory-scripted, where you are on the far end of the scale and I am closer to the middle and the certification people and other famous authors are on the other end?

/Toby

PS: How did you spend your spare time before you had comments turned on to your blog

[James' Reply: If all comments were like this, I would have more spare time. Good points.]

[James' Reply: On behalf of skilled and diligent testers everywhere, please accept my apologies. Indeed, a skill-free tester is not able to do much with a partly-baked product. My heuristic of instant release applies to me, testers who work for me, and testers who are like me, but not to every tester. It is something I'm suggesting that testers say to their programmer counterparts; something I hope we aspire to, in any case. I'd like to see no distinction between the testing phase and the programming phase. The whole concept of separate phases seems to me based on a myopic and defensive idea of what programming and testing can be.]

This topic creates a gray area for delimiting the end of development testing and the start of system testing.

We need to make sure that the acceptance criteria is clearly defined for everyone throughout our development effort so that the developers not only test the technicalities and architectural side of code (conformance to specifications), but also that their code implements the desired functionality for the end users (conformance to user needs and wants).

[James' Reply: I'm interested in where this need comes from, for you, since I have not experienced a context where this need exists. I find that acceptance criteria is not a pre-requisite for testing, but rather that testing is a process of refining acceptance criteria. By the end of testing, ideally, we know what we are accepting and why we are accepting it.

Also, as I've already written, I don't particularly need the programmers to do any testing. I appreciate it when they do, just as a paramedic appreciates when people on the scene render CPR to a heart attack victim, but don't delay calling me in because you feel like you should test. Don't delay. That's my message to the programmers I work with.]

This knowledge can be leveraged by jointly reviewing all documents directly derived from requirements: use cases, design specifications, functional specifications, technical specifications, test cases, and of course, the requirements specification document itself.

Developers must be completely aware of what is expected of their code, and to achieve this, cross-functional meetings can be organized before and after every phase of the SDLC to: prevent defect injection, identify defect predictability patterns, and minimize heavy reliance on quality control to find bugs.

By doing this, we are empowering the creators of every deliverable in the SDLC with the knowledge to produce the best quality interim products in terms of compliance to standards, and customer satisfaction (and yes, that includes code).

[James' Reply: In general, I don't find that quality software comes from extensive planning and documentation. I think that's because so much of the extensive planning is bad planning, and so much documentation is bad documentation. I suppose, like cocaine, it's possible to indulge in that stuff and not get in trouble, but mostly I just see people get in trouble with it. I prefer a more agile and incremental approach to achieve excellent quality software.]

We are together in this software development thing; let everybody do what they do best and remember that everybody else can learn something from it.

[James' Reply: I don't follow your reasoning on this. But anyway, have you considered the downside of total independence? There's always a tradeoff, don't you think?]

I’ve posted a further exploration of this from an Agile development perspective at:

http://www.extremeplanner.com/blog

Clearly, the context of a test done by a developer and a tester is different.

What I expect of a programmer is to do Unit testing. By that I mean the program should of course link and compile successfully, and perform what his/her change is supposed to do. That is when I would consider that the code “exists” and that I can “reasonably” work with.

If all that a programmer does is ensure that the code compiles successfully and hand it over to the tester, well, the tester’s life will be a nightmare.

[James' Reply: If my life is a nightmare, then I will talk it over with the programmer and we'll fix the problem. Working together on specifics trumps any general heuristic I can utter. However, I find myself more worried about the nightmare of a programmer whose reticence to share his work leaves me with too little time to do my work.] 

To me, a tester should look at testing in a bigger picture…as a system with all the relationships and integrations with other systems. I would not expect a developer to do this kind of testing for sure.

I guess it will help if you define what you mean by a “test”.

You have mentioned about being of service to your customer – the developer. How exactly are you helping the developer? For all you know, you will have the programmer get even more frustrated when a tester starts looking at a half baked product. In fact I have done this myself because of my curiosity to see how the product looks like before it is handed over to me. The moment I communicate my observations (informal or not), more often I get “Can’t you just wait till I hand it over to you?”.

[James' Reply: I appreciate this line of thinking. It's an important consideration. Part of the answer lies in the fact that finding bugs is not the only thing I do with a product. As I wrote in my original post, even a completely inoperable product can help me. It can help me prepare for highly productive testing, later on. Another part of the answer is that adjust the way I work so that I don't frustrate the programmer. You may not realize that I also am a programmer. I have some empathy for the difficulties of software development. My focus is on serving the programmer and my other clients, as well, by working in such a way that the programmer's productivity is maximized. So, if what I'm doing isn't helping, then I adjust what I'm doing.]

Of course it is going to be difficult without enthusiastic cooperation of the people who developed the product. However offering your services to test it before they have a chance will not get you what you want.

In my experience, most often I get into a confrontation with developers is when the testers due to whatever reason do not test a product the way it should be tested. As a result create a bunch of defects that a developer has to analyse, only to find that there is nothing wrong with the code.

No, the developer would not prefer a tester to test before he/she is “done” with it.

So what if your testing process is delayed, you can always do something else until a “reasonable” product is delivered to you. If not, inform the management the reasons why your testing is being delayed.

Oh well…just my 2 cents.

Chetty

p.s.: BTW, I did see your video on becoming testing experts. It was a good presentation and I am going to recommend my team members to watch it as well.

[James's Reply: Thanks for the plug. Bear in mind that what I wrote is not speculation, it's the way that I can and have worked with programmers, for a number of years. I push to get the software as early as reasonable, and that is generally earlier than most programmers are comfortable with, at first. As I work with them, I generally can make them increasingly comfortable with earlier deliveries. This has to do with charm and diplomacy and establishing my technical credentials.]

So what, you want your developers to email you each time they write a line of code?

[James' Reply: I love a comment that gets to the meat of the argument! Thank you!

The answer to your question is probably not. (Of course, if you write one line of code per day, and you do one build per day. Then I would like that build. You don't necessarily have to write an email.) What I mean by "exists" is that it exists in a form that I could reasonably work with and that you can reasonably part with. For one thing, it should compile and link. For another, the process of releasing to me and working with me should not cause you to interrupt your flow of thinking. You and I would work together to figure out what is reasonable. You will not hear me ask you to delay until you've tested it, however.]

Aside from wasting the developer’s time sending out emails every few minutes, there is no use ‘testing’ code which has nothing in common with what will be shipped (which is what many first builds are).

[James' Reply: I'm not suggesting that you waste your time. Please don't waste your time.

However, there is considerable use to "testing" code that has nothing in common with what will be shipped. I suggest that you should not try to decide for me what is useful to me as a tester, any more than I would presume to second guess your development process. But I'm happy to discuss with you the value that I get out of seeing things early. Our specific discussions, on a specific project, would trump any heuristic.] 

If you really have nothing better to do, find some education to go through or test other parts of the program (even if you think you have found all the bugs, I’m certain you missed some considering how many I’ve found that our test teams have missed). In fact, if developers send off code without running the most basic tests first, the most likely consequence is that they will break the entire product and you won’t be able to test a thing. Plus it is much more economical for developers to find bugs during development than for them to wait for the testers to find it.

[James' Reply: I don't find that it is necessarily more economical for developers to test, unless you are talking only about the simplest kinds of tests. One of the problems with developers testing is that many of them are bad at it, and most of them are uninterested. When I worked as a production coder, in the early 80's, I certainly was both of those things. Furthermore, I want to recommend, if you are a developer working with me, that you and I test the product together, at the earliest reasonable time (which will be very early indeed), on your own system. When I have done that with a developer, it has been fabulously productive.]

I’m not saying developers should share the sole responsibility for testing, I don’t think anyone is arguing that (at least for a commercial product, if I’m doing something on my own I’m not going to go out and hire a test team to help me). Sure, a second eye is needed to make sure the damn thing works like it is supposed to and not just how the developer thinks it should (in fact, that includes other developers as well, that is why we have code reviews). And no, developers should not pass of their code to the test team the day before the product’s release and expect the test team to just rubber stamp it. But that doesn’t mean the developer should pass of his code without first testing it all. He shares some responsibility for getting it to work as well.

[James Reply: You can take on all the responsibility you want. However, if you and I are working together, I am going to make it my mission to charm, cajole, bribe, and otherwise do whatever I can do to relax your iron grip on your baby so that I can do my job and help you do yours.

My suggestion may not work for you if you are working with low competence testers. Frankly, I wasn't aiming my post at developers, so much as at other testers. I'm trying to convince other testers to change their attitudes.]

[James' Reply: As a programmer myself, I am confused about why you believe this. I'm seeing words like "only" and "cannot" and "comprehensive", etc. used in such a way that imply that there is one and only one way to write high quality software. Have you tried doing it in other ways? Are you aware of the many styles and approaches that are available to you? Here are some honest assertions of my own:

My understanding of refactoring is adjusting software design without changing software functionality. I can refactor code without any testing at all. Testing is certainly not a prerequisite for refactoring. (I'm not saying I would prefer to do it without testing, but neither is there any specific testing requirement for the process)

Show me a "comprehensive" set of unit tests, and I'll show you ten more tests, off the top of my head, that aren't in your test suite. I'll show you a hundred more. 

A developer doesn't need to do any testing, at all, if someone like me is working directly with him. Again, I'm not saying that developer testing is bad. Personally, I appreciate developer testing, but I consider it optional. To make it a demand or expectation endangers the very idea of skilled independent testing.]

Testers are people who assure that the functionalities written by coders really meet end-users’ needs (maybe different from what coders think they have to code.) In addition, testers has to assure that all the non-functional requirements such as security, performance etc. are met properly.

[James' Reply: That looks to me like a weak idea of what testers are. I prefer a more potent idea.] 

Saying that, IMO, in good software projects, both coders and testers much be responsible for testing.

[James' Reply: Are you also prepared to say that coders and testers are both responsible for coding? If not, why are coders supposed to encroach on my job when I'm not allowed to encroach on their job? If so, aren't you really saying that you want everyone to be a programmer, and some programmers do more testing than other programmers?

The trap, here, Nguyen, and it's a pretty big trap, is that expecting coders to test either requires coders to develop testing skills, or it requires coders to test incompetently. In my experience so far, coders who test do so without much competence. I don't mind that, as long as they don't start lecturing those of us who study testing about how it should be done.]