In the circumstances I am thinking of, which I get to deal with fairly routinely at the moment, the dev team gives the BA the spec, which says “The product shall…”. The BA writes the test cases based on the spec, which read “The tester shall test that the product shall…”, and then hands the test cases to the tester, whose mission is to validate the product relative to the spec vis a vis the test cases. Anything beyond that is seen as wasting your time and not doing your job.

I do not feel in any way bound by the limitations of that mission. First of all, in my experience, when a B.A. says “the tester shall”, that’s just a suggestion (and usually an inadequate one) of what I shall really do. From most B.A.’s, that stuff is on the level of “the tester shall show up for work”. My job, as I see it, is to take the suggestion and run with it, since (without considerable experience) they have no idea of what I can do; they’re bound by their imaginations. I don’t have to be bound by their imaginations.

Now: I’d bet you have this attitude too, and I’ll bet I understand the reaction that you may have got. I got that kind of reaction from one client. I discovered a fairly spectacular problem, and the in-house tester was indignant. “You’re supposed to be testing [something else]!” What he really meant was “I wouldn’t have thought to do that, and I’m surprised that you did.” My response was twofold. First, I had exposed the bug by throwing the program some extreme data on the way to my primary destination, so in fact I was testing the stuff I was supposed to. But in addition, I said, “Okay, I can see how you might believe that. But tell me: if I happened to find a problem like that at some point while I am testing the stuff I’m supposed to be testing, would you be interested?” Long pause. “Well…” Long pause. “I guess.”

What you were experiencing was that a spec was being reverse engineered to address a set of features already believed to work well, not known to work well. If I’m in such a circumstance, and I reveal some information that might lead to a delay, I remember this: I neither put the problem in, nor am I responsible for the decision to delay or not. Gathering information about the product is my responsiblity; what the team decides to do with that information–delay or release–is the decision of the project owner. This doesn’t have to be an adversarial process. Would the developers prefer to release the product with bugs that they don’t know about? I haven’t yet met a developer who can’t be convinced that I’m trying to make him look good.

[James’ Reply: Well said!]

I believe that is exactly the point. In the circumstances I am thinking of, which I get to deal with fairly routinely at the moment, the dev team gives the BA the spec, which says “The product shall…”. The BA writes the test cases based on the spec, which read “The tester shall test that the product shall…”, and then hands the test cases to the tester, whose mission is to validate the product relative to the spec vis a vis the test cases. Anything beyond that is seen as wasting your time and not doing your job.

What was really going on here (IMHO) was that a spec was being reverse engineered to address a set of features already known to work well, and the dev team was defining the mission so narrowly as to avoid any possible delay to their release that we may cause. Quality was not the overriding concern, schedule was. In the process of changing that, relationships have become adversarial, and that is the source of some of the frustration that I am sure can be sensed in my posts…I don’t like it when testers and developers talk to each other with suspicion and disdain. But as long as the test leadership chooses the moral high ground of “Quality for qualities sake” (a whole different discussion), and the development leadership gets whipped for missing delivery dates and deadlines, we continue to muddle along in our little rut.

The more I observe this situation, the more I believe the problem is one of leadership…the true mission has not been defined at the proper level and communicated down to those tasked with carrying it out. This leads to much frustration, which, by the time it gets to the individual testers level, at least LOOKS like we are being told to test strictly to the spec, which we inherently know is not good testing.

Sincerely,
David

[James’ Reply: I believe it is possible to pretend not to draw inferences about the spec, but I think that would just  be pretense.] 

[James’ Reply: Let’s untangle some issues that you may have confused. The specification is not the mission. The specification is not the test plan. Do not conflate the specification with these other things. The specification does not tell you what to test or how to test. The specification merely raises important questions. If you want to find important problems quickly in that product, you must perform testing that addresses those questions.

To fulfill my mission as a tester, I might not test what is referred to in a particular spec, or I might test everything, or I might test other things not in the spec. I do what fulfills the mission. If the mission is merely to touch the things that are mentioned in the spec, and not to draw inferences about them so as to address the risks that my clients are concerned about, then I will probably miss some important problems. My client may come to regret that they insisted on such a limited mission.]

If, within the freedom of the constraints you listed, you then have the luxury of time to go back and test more than you were tasked (I realize this is one of the constraints you described) then that is great, and exactly what I would recomend.

[James’ Reply: I don’t know what you mean by “tasked”. If its covered by the mission, then we do it. If its not covered by the mission, then we probably don’t worry about it. However, a tester’s mission often includes such ideas as “find important problems, please.” This will require you to go beyond, and sometimes far beyond, the handful of things mentioned in most specifications I’ve seen.]

So the spec is the starting point, but in some cases, even after considering your response, it may also be the ending point, and I believe your response shortchanges that a bit.

[James’ Reply: A specification document is not, and cannot, ever be an ending point. It is a simple impossibility. As soon as you read a specification; as soon as you think about it at all; you extend it. That’s because every specification document involves assumptions and requires background knowledge to interpret. It evokes your imagination.] 

This is where I was 3 years ago, and in that time, we have managed to expand the constraints you outline, with the biggest change being in Project Culture. The real problem was that our Mission was badly defined, but we had no base of power to change that. By simply finding things we weren’t necessarily asked to find, but that were important, we were able to change the cultural perceptions of our role, and this led to a redefining of our mission. To your point about being careful, we did make many people unhappy along the way, but ultimately their unhappiness was based on the fact that our success was due to their not really supporting the organizations mission properly, and you could write books on that all by itself.
All of this is interwoven very tightly, but at the bottom of the food chain where most of us testers get to spend a lot of time, sometimes life does indeed begin and end with the spec and those other constraints are simply outside of our sphere of influence.

[James’ Reply: One thing that is never outside your influence is your ability to misunderstand a specification. And just as no one can prevent you from misunderstanding the spec, no one can prevent you from understanding it excellently. No one can prevent you from having great ideas about testing a product based on implications you see in the spec, or from any other source.

I’m not talking about flouting your mission. Your mission binds you. The specification is just food for thought.]

I feel i have some clear views & ideas about testing our websites for usability.