But couldn’t the school of testing that you follow depend on the software you are testing.

For example, If I am testing a bomb, might I want to follow an Analytic School’s approach. Making sure I have precise and detailed specs and the bomb works accordingly? Or would you say a context approach might be yes you need to test the specs to the last detail, but hey, maybe you should test what happens if you touch the red button not listed in the specs?

If a Quality School asks the question “Are we following a good practice?�. Is it a bad thing to ask evaluate the defects that were missed and why? And in the end we can determine if we followed a good testing procedure? But if we also learn from our mistakes, would that be more of the context school.

Maybe I have too much of a simple mind to get all this philosophy. Maybe I should just say I see software therefore I test…

[James' Reply: I think you are confusing practices with schools. Any school could use any practice.]

Upon deploying code to an environment that inherently differs from another, we are shocked with the lack of conformity to our specs. We came to realize that a number of random accidents in the software development, quality assurance, deployment, and management of systems meant that there were several combinations of systems that resulted in poor achievement of the original objectives.

It reminds me of Edward Albee’s observations that a play might have to be author-proof, actor-proof, and audience-proof in order to succeed.

[James' Reply: Good point, and well said.]

Someone already familiar with a product will be able to tell you whether the documentation or Help is correct, incomplete etc. However only a new user to the product (and thus oblivious - assuming they haven’t received any prior training) will be able to tell you whether the documentation or Help does just that and actually helps them.

Depending upon the experience your new user has of your type of product (is this context specific - they might know how to use your competitor’s product?) or whether they are totally new to the product type, the way in which they use the Help will be different.

I believe this is true with usability testing. If I am testing a website that has a diverse audience from many different backgrounds, I will want testers that know the least about the Web or using a computer. In this case, knowledge of the context would be a detrement to a tester because it is difficult to pretend not to have knowledge once you have it.

Another example would be beginner tester’s luck. I’ve seen testers with a fresh perspective on an application find defects that the hardened, knowledgable, seasoned testers didn’t find in 5 years of testing it. In these cases,puting a new tester in front of the application with no context whatsoever might be more beneficial than bringing that tester up to speed on the context.

[James' Reply:  I like these examples. In both of them, notice that the tester is acting as a tool in someone else's process machinery. In such situations, obliviousness is no sin, and may be a virtue.] 

[James' Reply: Hi Keith. That's right. I'm a tester, not the project manager, not the CEO, not the programmer. If someone wants to follow XP, that's fine. I'll adapt to that.

To be context-driven is not to be uninterested or to have no opinion about what would be better working conditions. It's just that I know that I'm not in charge. And it's not only me. Realize that the project manager, etc. also have some serious problems if they expect to drag an unwilling or unskilled team along with them on an Agile adventure. Agile development, as you know, is a personal discipline.

I've seen a description of context-driven testing on the web, written by no one I've heard of, which calls the Context-Driven school a kind of agile testing. While there are similarities between them, they are absolutely not the same.]

One other example.
Context-driven attitude would not be a good example in situations where the ability to think is reduced. Two examples come to mind.

A) My 5 year old does not have the ability yet to distinguish context as well as I do. While I train him to recognize context, I still prefer that he would follow our specific instructions in most cases.

[James' Reply: But you expect-- and you need-- your son to adjust to context in all situations where you have not specified a program to guide his behavior. You expect, for instance, that he will not follow instructions given him by strangers on the street, unless the stranger is a police officer, etc. Furthermore, your son, like all children, learns from playing with rules.

With regard to behavior that you intend to control, you take responsibility for the quality of that outcome, that's what makes it reasonable. That is one of the big justifications for context-oblivious behavior.]

B) I have noticed for myself that my ability to think gets reduced in extreme stress situations.
When there is a fire in my office building I will follow the evacuation guide. While many other options might be better to evacuate the building a standard evacuation process would avoid chaos.

[James' Reply: The purpose of training to deal with emergency situations is to maximize the chance that reflex behavior, which remains available to us under stress, will also be effective problem-solving behavior. It's essentially context-specific training. However, another aspect of dealing with emergencies is to recognize and deal with important context variables.

Recently, my older brother was in a situation where the emergency checklist for his MD-80 airliner advised him to shut down an engine that instrumentation reported to be on fire. He and the other pilot decided that it would be more dangerous to follow that instruction than to skip it, since a one-engine landing in a heavy airplane at high altitude on a hot day is itself quite dangerous. They made this decision because there were no "secondary indicators" of a fire, the assessment of which is a matter of skill and insight. The airline convened a board of inquiry which questioned the pilots deviation from the approved procedure, but ultimately approved their behavior. That's why we have pilots in airliners!]

[James' Reply: I don't see how automatic transmission is context-oblivious. It seems that you are saying it is context-oblivious because the driver is not involved in the transmission decisions. The problem I have with that is that the driver is also not involved in the transmission at all. If you are doing something and you don't know why you are doing it that way, that's context-oblivious. But if you aren't involved at all in a task that is happening, that's not context-oblivious, that's task-oblivious. For instance, in all kinds of modern vehicles, the driver also does not control valve timing.

Can you think of a task that the driver does in a certain way, but might not know why?]

‘Any decent context-driven thinker can cite at least three scenarios where taking the context-driven attitude would be a bad idea. Can you think of them?’

Offhand …

1) Some companies care more about being stable, predictable, and repeatable than they care about being _good_. Some have even made a reputation offering products and services that may not be great, but are the same, every time. Most Brand-Name American Fast Food comes to mind. No matter where you eat it, a Big Mac is the SAME - every time. Trying to examine the situation and determine the best thing to do is not what the manager wants you to do; he wants you to follow the operations book.

[James' Reply: That argues for context-oblivious. The reason it's a reasonable argument, to me, is that someone else is taking responsibility for the quality of the results.] 

(#2 is not politically correct, but it is my life experience …)
2) While the American Front Line Combat Soldier is one of the best trained in the world, there are times when the solider needs to obey all orders immediately and to the best of his ability, because there simply is no time to think. Two things that come to mind are when the squad leader points and yells “Run!” (which probably means incoming mortars or artillery) or the invasion of Normandy.

Actually, those situations are rare; the American fighting soldier is trained to think and act quickly, to be situationally aware. That said, there will always be a time when an NCO makes a statement with authority and it needs to be obeyed unquestionably or people may die. (This is where trust comes in)

[James' Reply: This is a good example. Though there are exceptions (a soldier is not required to follow an unlawful order, for instance), this is also a situation where it may be correct to be context-oblivious. The non-commissioned officers are probably context-specific. The commissioned officers probably need to be context-driven. The great generals of history certainly were.]

3) Sometimes authors and speakers need to use a bit of hyperbole when making a point. For example “‘Any decent context-driven thinker can cite at least three” sounds a lot more emotionally appealing that “It has been my experience that any decent …” or “All of the decent context-driven thinkers who I have met have …” We see the point. (Ok, that one was for grins.)

[James' Reply: I wouldn't call that hyperbole. I was not trying to make an generalization about the whole world based on my experience of a piece of it. I was speaking my opinion as to something that defines the attribute of "decent" as it pertains to a context-driven thinker. Since I didn't identify it as my opinion, I guess you could say it's a context-imperial statement, i.e. "if you don't think this way, I suggest that you start thinking this way." Is it a good thing to do that? I don't think it's necessarily bad, but neither is it necessarily good.]

4) If someone else is falling into a trap of one of the other schools, engaging them in a discussion may fail. What you can do that point is to take a logical position from a different context and adopt it into the current problem. The other person may immediately recognize that you are making a mistake, criticize you for it … and have an “aha� moment.

[James' Reply: That's an interesting example! Non context-driven behavior as reductio ad absurdum.] 

5) Sometimes I like to do context-oblivious manual labor to relax my brain.

[James' Reply: Not really an argument, though, is it? No matter how sleepy I feel, if I'm driving my car, I'm not allowed to go to sleep before I park it. Just because you don't feel like doing a good job doesn't necessarily release you from the responsibility of doing it.] 

6) There are some things to which we are context-specific, like “I am breathing oxygen�, that, most of the time are not worth investing our time in thinking about. Of course, throw me in a lake and bind my legs, and my thinking will change.

[James' Reply: Often we are oblivious to that. Recently, two college students died when they crawled into a large helium-filled balloon. They didn't know that helium, although non-toxic, does not solve the same problem that air does. But if you do SCUBA diving then it breathing becomes context-specific. I know why I am puffing on this regulator. I am trained in it.

Clearly, Matt, you are onto a method, here. All you have to do to imagine ways in which it may be good NOT to be context-driven is to imagine how the OTHER attitudes might sometimes be a good thing.] 

[James Reply: I leave that definition to you.]

Consider an automatic transmission - changing gears automated based on some defined gear range values (best practice, huh). While it does not consider the context it reduce fuel efficiency and power compared to manual. And still most cars sold in the United States since the 1950s have been equipped with an automatic transmission. Is Europe a context ?

[James' Reply: Are you saying that it increases or decreases fuel efficiency? In any case I'm not sure what that example indicates, because it is easy to argue that automatic transmission is good or bad, depending on context. I prefer to drive exclusively manual transmissions (it's more fun), but my car is an automatic due to my wife's fear of having to start the car on a hill. We both can drive automatics, hence that is what we have. Fuel efficiency doesn't matter at all, to me.]

I also consider myself to have a context-driven attitude. However I like testers who cultivate the skill to succeed in one stable corner of the world and as a manager try to give them tasks to suit their own skill and seldom - to expand their world.

[James' Reply: What's better about that kind of tester than the kind who adapts? Are you saying that you would not hire someone like yourself?]

I’ve recently written a testing chapter for methodology guide for our company. As I’ve blogged previously I don’t like to document what and how to be done. Still I accepted this challenge. In this chapter I used a lot words like “By default, it is a common approach to, it is recommended to, typicallyâ€? to describe what someone could call the best practice.

[James' Reply: When I write articles, I avoid using those phrases without also talking about alternatives. If it's the default to do X, why is that the default, and what other choices are acceptable? I don't say "X is recommended", I say something like "I recommend X, in situation Y, because Z." I try to talk about problems, solutions, and the problems those solutions could create.

However, writing articles is different from writing a methodology guide for a company. When I do that, I prefer to record policy and reference information. I don't try to write down instructions to convey skill and knowledge. That is much better conveyed through person instruction and supervision over time.]

Sometimes I describe few different practices which “depends on software type/test type/project phase/etc.” . And stressed out in the most significant places that “It is up to the Project Manager to decide…” . While doing this – I don’t represent the context-driven school while documenting the best practices, do I? And still there are certain benefits of having the document. If only you avoid two things: 1) obey the document 2) hope to improve process by changing the document

[James' Reply: I find there's art and discipline to writing a guide to methodology. It might help us to have a context-driven methodologists guide to giving advice. Maybe I'll write one.]

bq. I think a general case probably applies - it is a mistake to (try to) explain things in a context-driven way when doing on-the-job training. Just tell the person what to do, and maybe drop some hints on the context. If they have the aptitude to learn, then they will take the initiative and follow up on those hints on their own.

Not to rake you o’er the coals, Steve, but I think it still depends. A couple of quick examples:

If I am running a business (such as a fast food franchise), rather than conducting an apprenticeship (such as once was the case in craft guilds), the balance of time spent explaining context should be different. If I am explaining “don’t cross the street without holding someone’s hand”, I expect how I explain it will differ if my subject person is 2, 22 or 95 years old.

I prefer apprenticeship contexts to follow-the-rules franchise/Harvard MBA ones, especially when the mentor says “just try it my way for {x} period and we can discuss what we both see as an outcome”.

And yes, many apprenticeships are-were run as “just follow the rules” because world-lessons (when actually learned) sink in in a way that word-lessons do not. I think that’s at the heart of your approach; am I right? I’d add that another reason is that people are neither perfect learners nor perfect teachers, and time and attention are limited resources.

Classical 20th-Century business school taught managers to make “humanresources” (I always run the words together to remind me the term is-was Newspeak) as fungible as possible. “There are no irreplaceable people”. True, but but but… not every business context is an assembly line. I’m sure you agree!

I took high school physics from a teacher whose attitude toward my problem-solving was “That’s just wrong. Points off for not being right. Full stop.” It was not through any effort on his part that I took Physics as my major in college. If Feynman had been my instructor, I have the feeling that the tone would have been more like “Hmm, well, I can see where you might think that, but here’s what we actually see, so the effect you’re talking about is probably vanishingly small, and nobody has reliably produced it yet. So we don’t find your approach useful.”

Big difference, same domain (”teaching physics”); different (meta?)context.

[James' Reply: Good point. That is, actually, one of the contexts in which I discourage context-driven thinking: the context in which you are, as yet, oblivious to context, and you are under the supervision of someone who take responsibility for the value of your work.

Remember the line from The Hunt For Red October?

]

Can I consider context driven thinking as collective thinking of the all other schools of testing as described by Bret with context assuming the prime significance?

What is your opinion about following equation?Context Driven thinking = Analytic thinking + Quality Thinking + Factory thinking + CONTEXT (with first three components of varying proportions depending upon the context and in some cases, one or more at ZERO)

Shrini

[James' Reply: No, the schools are independent belief systems. Each stands alone. I don't do "factory thinking." I might select a factory-like method, but not for the same reason as a Factory school partisan.

Do not confuse a school with the practices it tends to use. A Christian and a Muslim and a Hindu may all drive the same kind of car, but that indicates nothing about the overlap of their belief systems.]

Install testing of a Windows application versus testing a website. Here’s the situations:

1. Install testing. When testing an install, it’s important to have and maintain a known state in regards to what dlls exist on a workstation before and after the install assuming the install includes dlls and assuming the testing includes checking the date/time/version of the dlls on the PC. The context I am specifically envisioning is a Windows application that uses shared MFC dlls not just proprietary dlls.

2. Website testing. When testing a site it might be important to know how the website functions with Windows Update turned on. The Windows Update may impact the site’s functionality and skipping an update might be skipping what customers will experience. Better is not updating automatically but updating at decided intervals. What I’m suggesting is turning Windows Update off and only after a review of each update and knowing what is being updated accepting the update and then test the relevant or potentially impacted areas of a website. Of course, what is supposed to be impacted versus what ends up being impacted is not always the same. Different story.

Maintaining state versus not maintaining state. Switching out dlls during testing an install would be reckless but then not testing with Windows Update turned on may not simulate real world usage when testing a production website. I’ve done a fair amount of install testing and the state of a test PC is a condition to be mindful of … I’ve blogged about the state of PC ….

In sum:
To test with Windows Update on would strike me as poor practice for install testing and applications where MFC dlls are involved. (Unless of course, you want to test on a “dirty” and uncontrolled PC to see how the install handles … see how the context shifts and I have to keep thinking about what it is I’m doing, what it is I want to accomplish.)

To test with Windows Update off for a website strikes me as being too clean to simulate real world experience. (But then knowing how the site functioned before and after Windows Update is good knowledge to have … again what is the context, what is the goal …)

Two different contexts. Two different answers. Or more.

[James' Reply: What a rich example. There's enough detail here for us to have a conversation. You are also signalling me, with an example like this, that you A) have a command of details and B) have relevant experience with these details. To be context-driven requires that we routinely speak this way amongst ourselves and with our clients.]

I think a general case probably applies - it is a mistake to (try to) explain things in a context-driven way when doing on-the-job training. Just tell the person what to do, and maybe drop some hints on the context. If they have the aptitude to learn, then they will take the initiative and follow up on those hints on their own.

What I translated from your response (after filtering out the ‘you’re an idiot’ innuendoes) is to not limit the testing to just meet regulations and that testing heuristics is needed in conjunction.

I have no experience testing missiles, airplanes, etc. I attempted to imagine what it would be like if I was thrown into such a situation. I would be able to apply testing heuristics, but due to my inexperience, I could possibly miss an important function or what could also be known as a regulation. I do understand that testing cannot be limited to these regulations but do not understand why we should do without regulations. Perhaps your experience has shown you that not-so-bright people are creating these regulations and they are worthless anyways.

I do appreciate your initial response and will continue to keep studying to hopefully someday be a great free thinking/heuristics software tester. Don’t count me out just yet!

[James' reply: I'm pleasantly surprised that you responded so graciously. I may have misjudged you from your first comment. If that is the case, please accept my apology. There are specific reasons why your comment provoked my response, however.

Let's start with what a heuristic is: A heuristic is a fallible method for solving a problem or making a decision. ALL testing practices and regulations are heuristic. In other words, there is no "guaranteed, just turn the algorithmic crank" way of testing that will find every important bug. When we are talking about which practices to apply, the discipline that matters is the discipline of working the problem, not the discipline of following instructions. Only once we have decided on a course of action, it's relevant to speak of following the plan.

One reason I reacted strongly to your comment is that your comment implies that someone else, wiser than you, has decided for you what you should do, and that your role is not to question, not to "think freely" as you put it. But this is not the case (unless you meant to say that your boss has set these regulations for you, and you must do as he says, for he is defining success for you). There are regulations in some contexts, true, but none of those regulations state that you are free to do bad work as long as you follow the regulations. They do not release you from responsibility to think. In other words, the regulations are also heuristics.

There is certainly no support for the claim that hazards would be created by not thoughtlessly following some testing regulation. I would claim that hazards are obviously created by thoughtlessness itself. Skill is the key; skill and knowledge. Not compliance.

If you face a regulation, then you must take it seriously. That may be an important aspect of your context. But there's no way to generalize from the mere fact that a regulation exists that it is good, or necessary, or not harmful.]

[James' Reply: Okay folks, pay attention, because I'm just going to reject every other comment like this that comes in. Since this was the first, I'll make my response, and I will just say it once.

High risk projects require better thinking, not worse. More thinking, not less. Free thinkers-- the ones who follow reason wherever it leads-- are exactly the people I need on intense and important projects. It is extremely irresponsible to imply that some government office or standard will protect you from having to do good work by giving you permission not to use your best lights to solve problems on critical projects.

You are intimating that there is an alternative to using heuristic approaches to test software. There isn't. In my experience, there are basically two kinds of people who say that we can mindlessly follow "regulations" that will lead us to glory: the kind of people who have absolutely no knowledge or skill, and the kind of people whose ambition is to profit from the ignorance of the first kind.

Which kind are you?]