Online Test Coaching and Collaboration

I’ve been watching television commercials about the amazing new world of technology since about 1997. You know the ones: people accessing the web on their phones, doctors diagnosing patients from 10,000 miles away, or people attending universities online, or meeting via video screens. The thing is, mostly this technology hasn’t worked. Mostly it has been hype.

Two years ago, I noticed that begin to change. The key moment for me was when my father had to ditch his seaplane on a remote beach, and I used my Treo 600 to access tide tables over the web to find out how soon we had to fix it before waves came up and battered the thing to pieces. Wow. I accessed the web from my phone and it actually helped me solve a practical problem.

These days I have a Blackberry, and it makes a great phone and a decent web browser. It also has great battery life, though still a little too slow on downloads and not quite compatible with every kind of website.

Video conferencing and general online collaboration remained impractical. But recently, I have experienced a collaborative revolution:

  • My son persuaded me to install Skype.
  • I upgraded to MSN Live Messenger.
  • I got a webcam.
  • I subscribed to GoToMeeting and GoToWebinar.
  • I started using Google Calendar and Google Docs.

Though still not fully realizing the promise of all those advertisements about how technology makes the world work better together, these tools definitely go a long way.

Michael Bolton and I video conference on a regular basis, now. It’s reliable and fast. It really seems to bring us closer as we develop our training materials. David Gilbert demoed his latest version of TestExplorer while we shared screens with GoToMeeting. I’m giving a webinar to a class at Calgary University, this coming Saturday, from my home in Virginia. I’ll be using Skype for the voice feed.

Just in the last six months, I have begun coaching testers online and collaborating with colleagues with unprecedented clarity. I feel almost like I’m in a cubicle cluster with my best buddies. (To me, this is a good thing. I like being interrupted.)

The capper was a few weeks ago, when Pradeep Soundararajan, in India, tested a product on my computer in Virginia, while Grig Melnick, in Calgary, looked on. We Skyped to the GoToMeeting conference bridge (not free, but not expensive by U.S. standards) and shared my screen. Apparently, we could have added another 8 people to that process with no degradation of performance.

There is sometimes a voice quality problem when I use Skype at the same time as GoToMeeting. Otherwise, I’m happy. The thing I still don’t have is a viable persistent secure online collaboration zone where I can share files and co-edit documents. Google does some of that. MSN Live Messenger almost does it, but its file sharing is still pretty much a joke. Wikis make my skin crawl.

What Difference Does it Make?

For one thing, I’m developing new testing exercises much more quickly, now. I’m doing more of them as chat sessions, too, which means I can save the results for posterity. Whenever I have an idea I can bug one of my friends to try it with me online.

I’m adapting my class for teaching online. This is not easy, because some of my exercises are very interactive and involve physical props, but I’m getting serious about it.

I’m strongly considering offering tester coaching as a new service. The way it would work is a company would pay a small monthly subscription fee for its testers to have me (and eventually a team of other coaches I call the Students Of The Craft) available via instant message, Skype, and GoToMeeting. That way, I can actually hover over their virtual shoulders as they show me their products and their test documents. I’ve done this already with a few testers. It seems to work pretty well. It may also be a way to run an economically viable tester certification service that’s based on observing skilled testers at work, over time.

Imagine being a small company, yet able to call upon a coterie of technical specialists (such as rapid testers, performance analysts, etc.) to solve specific problems as needed. We have to figure out how to make it economical, but I think it’s within reach, now that the technology is routinely able to support it.

You may see me on one of those “ain’t technology great” advertisements, before long.

Context-Driven Methodology

Today I am writing about the Context-Driven school of software testing. This is a community that initially coalesced around Cem Kaner’s Los Altos Workshops on Software Testing (the LAWST meetings), in the late 90’s. I am a founding member of that community.

In the last two years, more people have joined the community of active context-driven thinkers, helping to develop the notion of what “context-driven” means for testing. I feel like we have a critical mass.

Schools of Thought

In this context, “school” relates to this definition from Merriam-Webster’s Dictionary:

4 a: a group of persons who hold a common doctrine or follow the same teacher (as in philosophy, theology, or medicine) the Aristotelian school; also: the doctrine or practice of such a group b: a group of artists under a common influence c: a group of persons of similar opinions or behavior; also: the shared opinions or behavior of such a group other schools of thought

This definition is a good start, but we are referring to a stronger sense of school: a paradigm; an ontology; a way of organizing, experiencing, and thinking about the entire field. In other words, in the way I use the term, it’s not possible to belong to more than one school of testing at a time. If you tried to be in two schools at once, the school I think you would actually belong to would be a new school called Confused.

The value of the school concept is that it allows us to make progress in our work without constantly having the same fruitless arguments with people who are pursuing a fundamentally different view of the field.

I see the testing world in terms of six major schools of testing. These include the “breakthrough� schools of Analytical, Control, and Test-Driven Design, and the “pragmatic� schools of Oblivious, Factory, and Context-Driven. The breakthrough schools generally hope to reduce or eliminate the need for independent testing. The pragmatic schools generally accept the idea of dedicated software testers and deal with the problem of how testers should get the job done.

To the best of my knowledge, only my school has named itself. Bret Pettichord, Cem Kaner, and I, came up with the names for the other schools. More on the first five in another post, right now I want to focus on the world from the Context-Driven school perspective.

The Context Driven Basics

Cem Kaner came up with the term “context-driven”, in 1999. Cem and I drafted the principles of the Context-Driven School in 2001. Context-driven is all about people solving problems.

“Context-driven” means to match your solutions to your problems as your problems change. The principles of my particular flavor of Context-Driven methodology can be found here.

Consider this rant:

Stop memorizing things people say are solutions, then applying them without understanding how or if they actually solve the problems you face.Don’t pretend to solve problems– really solve them. Don’t solve pretend problems– really solve real problems. And you know what? Those problems are going to shift and change, sometimes day by day or minute by minute.

If you think you can do good work by following some guru’s scripted Best Practice, you might as well blindfold yourself, take a dozen sleeping pills, and drive the Daytona 500. But the nice thing about driving blind in the Daytona 500 is that you know when you’ve crashed and burned. The hard thing about software testing is that you can do terrible work for years and not realize it until some uber-bug that you could have and should have found nails some poor user right in his “GUI components.” Even then, you’ll probably just blame the developers for putting the bug there in the first place.

If you want to take responsibility for the quality of your own work, practices won’t save you: what saves you is your skill, plus your ambition to use your skill. I’m not saying you should ignore practices, nor should you hate them. Just don’t worship them.

To learn to test is to learn to solve testing problems in every situation (we call them contexts) that you encounter. If you do that, your process will unfold naturally, your methods will be fluid, your practice balanced and flexible. So ask yourself, do you have the skills you need? Ask yourself, is your testing fitting your context?

(That’s the end of the rant.)

I like ranting, sometimes. In a rant, I can bring the world into sharp focus. I can say bold, definite things. Unfortunately, the rant above, while advocating the context-driven attitude, is not itself an example of that attitude. Do you see why?

Let me explain…

Four Attitudes Toward Context

When I say “context” I mean the totality of a situation that influences the success or failure of an enterprise. If I want my testing to be successful, I need to behave, somehow, in harmony with the aspects of my world that make testing success possible. For instance, I’m not going to introduce the test technique of cause-effect graphing into the context of a team that has no idea what that means or how to do it, and no time or resources to figure it out.

Everybody copes with context. I see four interesting flavors of coping:

    1. Context-Oblivious

To be context-oblivious is to be unaware of the relationships between the way you work and the many aspects of the context in which you work. A context-oblivious practitioner is aware of doing things, but can’t tell us why it’s done that way. This is typical of novices, of course, but also of experienced people who do not like to think about methodology (a methodology being a system of methods; a method being some way of doing something).

Completely context-oblivious testers simply don’t read blogs like this. What would be the point?

    1. Context-Specific

To be context-specific is to be adapted to a context and to know that you are adapted. You know what problems you face, and you solve them. But those problems don’t change, and therefore you don’t need to adjust your solutions.

Most of us Americans would say we know how to drive a car. When we say that we are thinking of a particular context: driving on ordinary public roads in ordinary weather in ordinary cars. Any ordinary car driver in America would know that our driving skill in that context probably does not carry over to driving a race car in the Daytona 500 (even without a blindfold). Still less does it carry over to “driving” an airplane, even though there are many elements in common between airplanes and cars.

I don’t need to learn how to drive a race car, because I’m not in that context, and I don’t even need to learn how to drive in unfamiliar situations, because I simply don’t drive in unfamiliar situations.

It’s easy to believe that being context-specific is good enough, but beware: What if you suddenly lose half your team, or part of your testing get outsourced, or features of your product change radically, or you have to adapt to use a new tool. You may not have the option of adapting to one context only.

    1. Context-Imperial

To be context-imperial is to keep your solutions the same while striving to change the context around you to fit those solutions.

A classic example is when a tester refuses to test without a “complete specification” instead of learning how to test with or without a spec.

Another classic example is the typical testing consultant who asserts we should all use some Best Practice he has in his mind. When it doesn’t work, he says we must not be using “engineering discipline” to follow the golden path to success, or maybe we don’t have “upper management support” to force us to do the right thing. The solution can’t be wrong, so the context must be.

A modern example is the Agilist who insists that test-driven design is the only good way to write software, and the all software must therefore be written in an object-oriented language, and that 100% of all tests must be automated, because otherwise… well… it’s just not Agile! Kent Beck has spoken! Don’t anger the gods of Agile!

It’s the context-imperialists who bug me most. They are like doctors who prescribe medicine to patients they’ve never examined.

    1. Context-Driven

To be context-driven is to behave as if there is literally no such thing as a best practice. All we can hope for is good practices in context. Context-driven methodologists are very interested in learning about practices. Then they decompose and recompose them to fit the situation at hand.

To summarize the attitudes of practitioners…

  • The context-oblivious simply follow convention.
  • The context-specific cultivate the skill to succeed in one stable corner of the world.
  • The context-imperial insist that the world change to suit their own skill.
  • The context-driven cultivate the skill to adapt to a changing world.

These Attitudes Are Not Mutually Exclusive

I said the four types above were attitudes, not schools. (Note that I’m using the same label “context-driven” as both the name of an attitude toward context and the name of my school of testing thought which is centered on that attitude.) That’s because these attitudes, unlike schools, can mix together.

Even the most ardent context-driven methodologist (and I suspect I am that person) cannot be aware of all the aspects of context that matter in every situation. For instance, if a tester is secretly planning to sabotage the project, I don’t necessarily take that into account when planning that project. In that respect, I am context-oblivious.

I am also partly context-specific, because all my thinking about testing is colored and biased by the kinds of projects I have worked on in the past. Since I have done little military work, for instance, my ideas and calculations are probably different than they would be if I had grown up as a tester in that world.

I am also context-imperial, in some ways. If you want me to design a test process for a commercial product with the requirement that every single test be procedurally specified and placed into a test case management system, I would probably tell you that it sounds like a formula for bad testing and I would try to get you to change your mind. I’m not confident I can be successful in that context.

Despite all that, my self-image and my pride as a tester lies mostly with my belief that I can and I should be able to see a wide range of solutions to nearly any problem. That’s what being skilled in this craft of testing means to me.

Look Again at the Rant

See the unqualified assertions in the rant (e.g. “Those problems are going to shift and change…�)? When directed to a person in a context I know nothing of, they are the hallmarks of the context-imperial attitude. The context-driven way is to make speak in conditionals (“If your problems shift and change…�) and qualifiers (“…then you might find it useful to learn how to adapt your solutions to fit.�), or else stick to speaking about my own experience (“I find it rewarding to study the dynamics of testing problems in general, so that I can solve them no matter how they shift and change. This is because, as a consultant, I am asked to give advice in a wide range of situations.�)

If you look at my writing on this blog closely, you will notice that I use conditionals, qualifiers and personal experiences a great deal. Now you know why. It’s the price of belonging to the Context-Driven school.

Context-Driven Challenge

Any decent context-driven thinker can cite at least three scenarios where taking the context-driven attitude would be a bad idea. Can you think of them?

Suggested Best Practices?

A search on Google for the exact phrase “suggested best practice” turned up 9,230 hits, while “suggested best practices” turned up 30,300.

I wonder what the difference is between a suggested practice and a suggested best practice?

Is best a matter of suggestion? I don’t think so. What is lost by removing the word “best”? Like a bad hair piece, it just makes methodologists who use it look ridiculous.

Imagine if the Harvard webmaster removed the word “best” from this page: http://webmaster.harvard.edu/bestpractice Would it lose credibility? Or would it gain?

Question: How to Rapidly Test Maintenance Releases?

A correspondent writes:

“I have a test management problem. We have a maintenance project. It contains about 20 different applications. Three of them are bigger in terms of features and also the specs that are available. I am told that these applications had more than 1-2 testers on each of these applications. But in this maintenance project we are only 6-7 testers who are responsible to do the required testing. There will be a maintenance release every month and what it will deliver is a few bug fixes and a few CRs. What those bugs and CRs would be is not known in advance. Could you please suggest how to go about managing such kind of assignment?”

My first-order (direct) answer to this question runs about 3,000 words. It’s a slightly improved version of what was originally published in the TASSQ Quarterly, 9/06. It’s a bit too long for comfortable reading in blog form, so here it is as a PDF.

An Aside About Context-Driven Methodology

I also want to say that this PDF is an example of context-driven methodology. Responsible context-driven driven advice begins by inquiring about the context of the reader or specifying the relevant aspects of context in which the presented methods are believed to be helpful. The advice is then presented in a heuristic tone (“this might help”) rather than with an imperative tone (“you should do this”). I can use an imperative tone only when I am taking responsibility for the quality of the work; when I am the boss.

Part of the heuristic way of giving advice is to help the reader see reasons, causes, caveats, and alternatives. As a context-driven methodologist, I know my method ideas are tools, not facts. They must be interpreted and applied in specific situations by specific people. For the same reason, I see all methodology as embedded in a dialog. If I tell you “it is helpful to minimize documentation, because documentation is expensive”, I anticipate that you may reply with a question or a challenge, I try to make it easy for you to do that, and I try to be ready and able to answer you. Through the dialog, we all learn and develop better ways of working here and now on this project. In philosophical lingo, context-driven methodologists understand that each practitioner contructs (in a sense, invents) the craft for himself while immersed in a rich world of signs and signals that may guide us.

If you ever see me straying away from the context-driven path of methodology, I hope you will bring that to my attention. It is through the help of my colleagues and clients that I learn to do this better.