TestInsane’s Mindmaps Are Crazy Cool

Most testing companies offer nothing to the community or the field of testing. They all seem to say they hire only the best experts, but only a very few of them are willing to back that up with evidence. Testing companies, by and large, are all the same, and the sameness is one of mediocrity and mendacity.

But there are a few exceptions. One of them is TestInsane, founded by ex-Moolyan co-founder Santosh Tuppad. This is a company to watch.

The wonderful thing about TestInsane is their mindmaps. More than 100 of them. What lovelies! Check them out. They are a fantastic public contribution! Each mindmap tackles some testing-related subject and lists many useful ideas that will help you test in that area.

I am working on a guide to bug reporting, and I found three maps on their site that are helping me cover all the issues that matter. Thank you TestInsane!

I challenge other testing companies to contribute to the craft, as well.

Note: Santosh offered me money to help promote his company. That is a reasonable request, but I don’t do those kinds of deals. If I did that even once I would lose vital credibility. I tell everyone the same thing: I am happy to work for you if you pay me, but I cannot promote you unless I believe in you, and if I believe in you I will promote you for free. As of this writing, I have not done any work for TestInsane, paid or otherwise, but it could happen in the future.

I have done paid work for Moolya, and Per Scholas, both of which I gush about on a regular basis. I believe in those guys. Neither of them pay me to say good things about them, but remember, anyone who works for a company will never say bad things. There are some other testing companies I have worked for that I don’t feel comfortable endorsing, but neither will I complain about them in public (usually… mostly).

Mr. Langella Never Does it the Same Way Twice

This is from the New York Times:

Its other hallmark is that Mr. Langella never does the part the same way twice. This is partly because he’s still in the process of discovering the character and partly because it’s almost a point of honor. “The Brit approach is very different from mine,” he said. “There’s a tendency to value consistency over creativity. You get it, you nail it, you repeat it. I’d rather hang myself. To me, every night within a certain framework — the framework of integrity — you must forget what you did the night before and create it anew every single time you walk out on the stage.”

I love that phrase the framework of integrity. It ties in to what I’ve been saying about integrity and also what is true about informal testing: if you are well prepared, and you are true to yourself, then whatever you do is going to be spontaneously and rather effortlessly okay, even if it changes over time.

I often hear anxiety in testers and managers about how terrible it is to do something once, some particular way, and then to forget it. What a waste, they say. Let’s write it all down and not deviate from our past behavior, they say. Well I don’t think it’s waste, I think it’s mental hygiene. Testing is a performance, and I want to be free to perform better. So, I make notes, sure. But I am properly reluctant about formalizing what I do.

Doing your best work includes having the courage to let go of pretty good work that you’ve done before.

Behavior-Driven Development vs. Testing

The difference between Behavior-Driven Development and testing:

This is a BDD scenario (from Dan North, a man I respect and admire):

+Scenario 1: Account is in credit+
Given the account is in credit
And the card is valid
And the dispenser contains cash
When the customer requests cash
Then ensure the account is debited
And ensure cash is dispensed
And ensure the card is returned

This is that BDD scenario turned into testing:

+Scenario 1: Account is in credit+
Given the account is in credit
And the card is valid
And the dispenser contains cash
When the customer requests cash
Then check that the account is debited
And check that cash is dispensed
And check that the card is returned
And check that nothing happens that shouldn’t happen and everything else happens that should happen for all variations of this scenario and all possible states of the ATM and all possible states of the customer’s account and all possible states of the rest of the database and all possible states of the system as a whole, and anything happening in the cloud that should not matter but might matter.

Do I need to spell it out for you more explicitly? This check is impossible to perform. To get close to it, though, we need human testers. Their sapience turns this impossible check into plausible testing. Testing is a quest within a vast, complex, changing space. We seek bugs. It is not the process of  demonstrating that the product CAN work, but exploring if it WILL.

I think Dan understands this. I sometimes worry about other people who promote tools like Cucumber or jBehave.

I’m not opposed to such tools (although I continue to suspect that Cucumber is an elaborate ploy to spend a lot of time on things that don’t matter at all) but in the face of them we must keep a clear head about what testing is.

Who says ET is good for Medical Devices? The FDA!

In a new guidance document discussing the clinical testing of medical devices, the FDA includes a long section about the value of exploratory testing:

The Importance of Exploratory Studies in Pivotal Study

Medical devices often undergo design improvement during development, with evolution and refinement during lifecycles extending from early research through investigational use, initial marketing of the approved or cleared product, and on to later approved or cleared commercial device versions.

For new medical devices, as well as for significant changes to marketed devices, clinical development is marked by the following three stages: the exploratory (first-in-human, feasibility) stage, the pivotal stage (determines the safety and effectiveness of the device), and the postmarket stage (design improvement, better understanding of device safety and effectiveness and development of new intended uses). While these stages can be distinguished, it is important to point out that device development can be an ongoing, iterative process, requiring additional exploratory and pivotal studies as new information is gained and new intended uses are developed. Insights obtained late in development (e.g., from a pivotal study) can raise the need for additional studies, including clinical or non-clinical.

This section focuses on the importance of the exploratory work (in non-clinical and clinical studies) in developing a pivotal study design plan. Non-clinical testing (e.g., bench, cadaver, or animal) can often lead to an understanding of the mechanism of action and can provide basic safety information for those devices that may pose a risk to subjects. The exploratory stage of clinical device development (first-in-human and feasibility studies) is intended to allow for any iterative improvement of the design of the device, advance the understanding of how the device works and its safety, and to set the stage for the pivotal study.

Thorough and complete evaluation of the device during the exploratory stage results in a better understanding of the device and how it is expected to perform. This understanding can help to confirm that the intended use of the device will be aligned with sponsor expectations, and can help with the selection of an appropriate pivotal study design. A robust exploratory stage should also bring the device as close as possible to the form that will be used both in the pivotal trial and in the commercial market. This reduces the likelihood that the pivotal study will need to be altered due to unexpected results, which is an important consideration, since altering an ongoing pivotal study can increase cost, time, and patient resources, and might invalidate the study or lead to its abandonment.

For diagnostic devices, analytical validation of the device to establish performance characteristics such as analytical specificity, precision (repeatability/reproducibility), and limit of detection are often part of the exploratory stage. In addition, for such devices, the exploratory stage may be used to develop an algorithm, determine the threshold(s) for clinical decisions, or develop the version of the device to be used in the clinical study. For both in vivo and in vitro diagnostic devices, results from early clinical studies may prompt device modifications and thus necessitate additional small studies in humans or with specimens from humans.

Exploratory studies may continue even as the pivotal stage of clinical device development gets underway. For example, FDA may require continued animal testing of implanted devices at 6 months, 2 years and 3 years after implant. While the pivotal study might be allowed to begin after the six month data are available, additional data may also need to be collected. For example, additional animal testing might be required if pediatric use is intended. For in vitro diagnostic devices, it is not uncommon for stability testing of the device (e.g., for shelf life) to continue while (or even after) conducting the pivotal study.

While the pivotal stage is generally the definitive stage during which valid scientific evidence is gathered to support the primary safety and effectiveness evaluation of the medical device for its intended use, the exploratory stage should be used to finalize the device design, or the appropriate endpoints for the pivotal stage. This is to ensure that the investigational device is standardized as described in 21 CFR 860.7(f)(2), which states:

“To insure the reliability of the results of an investigation, a well-controlled investigation shall involve the use of a test device that is standardized in its composition or design and performance.”

This is what I’ve been arguing for a couple of years, now. If you want to test a medical device very well, then you have to test it in an exploratory way. This prepares the way for what the FDA here calls the “pivotal study”, which in software terms is basically a scripted demonstration of the product.

Yes, the FDA says, earlier in this guidance document, that it is intended to apply to clinical studies, not necessarily bench testing. But look at the reasoning: this exact reasoning does apply to software development. You might even say it is advocating an agile approach to product design.

Logging: Exploratory Tester’s Friend

I’m on a new project lately, working with a team at QualiTest. We’re testing a class III medical device. This is an exciting project, because for the first time I am aware of, formalized exploratory testing will be used to do such a validation. We will not rely on masses of procedural test scripts. I’ve been called in on this project because I created the first published formalized ET process in 1999 (for Microsoft), and created, with my brother Jon, session-based test management, which is basically a general form of that Microsoft process.

The QualiTest team consists of senior testers hand-picked for this job, who have regulatory testing backgrounds and an enthusiasm to use their brains while they test. On top of testing well, we have to document our testing well, and trace our testing to requirements. Automatic logging is one of the tools that will help us do that.

I am amazed at how crazy nuts some people get over documentation– how they sweat and shiver if they don’t have a script to cling to– and yet they don’t spare a thought for logging. Logging is great for testers, programmers, and technical support. Logging is automatic documentation. Sing the praises of logging.

I’m talking about function-level logging built into the products we test.

If you test a web app, you already have this (the web server and application logs, plus the use of a proxy to log locally, if you want) or would have it with a little tweak here and there by the programmer. For desktop apps, the programmer has to build it in. Here’s why he should do that right away:

  1. Instead of following a script written weeks or months ago by some over-literal, function-besotted and data-blind intern, the tester can think, explore, play, and maintain the thread of inquiry without worrying that you won’t know what you tested, later on.
  2. Instead of remembering what you tested, the product tells you how you tested it. Process the log with a simple Perl script, and you can potentially have an automatically generated test report.
  3. Instead of just wondering how you made that crazy bug happen, the developer can consult the log.
  4. Instead of asking the customer what he was doing moments before the crash, he asks for the log.

If logging is built into the base classes of the product, very little coding is involved.

This idea first occurred to me in 1993, after hearing from John Musa about how his telecom systems would “phone home” with data about how they were being used, but I couldn’t get a programmer to put logging into anything I tested until I was at SmartPatents in 1997. Since then I’ve helped several projects, including a couple of medical device projects, get going with it.

On this most recent project I was asked to create requirements to specify the logging. Here is the generic version of what I came up with:

1. Each significant action that the user takes shall be logged. (pressing buttons, touching screen objects, turning knobs, startup and shutdown, etc.) This provides critical information needed to demonstrate test coverage during validation, and improves our ability to meet and exceed regulatory requirements.

2. The results of any diagnostic self-tests or assert failures shall be logged.

3. Any function should be logged, regardless of user action, that causes a change to data, screen display, system configuration, modes or settings, communicates with other equipment, or produces an error or information message.

4. Everything that could be interesting and useful for testing, support, and system debugging should be logged UNLESS the event occurs so frequently (many times a second) that it poses a performance or reliability risk.

5. Each log event shall include at least the following information:
– Time stamp: For instantaneous events, time stamp (millisecond resolution). For events over time log the start and stop times by logging it as two separate events (e.g. “Event START”, “Event END”). Events that set a persistent mode or state can be logged as one event (“high security mode ON”) but the state of any such modes shall be automatically logged at startup and shutdown so that a complete record of that setting can be maintained over time.
– Event type ID: always unique to event type; IDs not re-used if an event is retired and a new event is created.
– Event type description: short, unique human readable label
– Event information: any data associated with the event that may be useful for customer service or assessing test coverage, this data may be formatted in ways specific to that event type.

6. At startup and shutdown, the current settings, modes, and confuguration shall be recorded to the log.

7. Any errors shall be recorded to the log, including the actual text of the error message.

8. Every type of loggable event shall be stored in one table in the source code or in a data structure accessible on the system itself, such as a header file, enum, array or resource file. This facilitates providing the validation and customer service teams with a complete list of all possible events.

9. The log format shall be in text form, structured and delimited consistently such that it can be parsed automatically by a third party tool. The data for each event should be on one line, or else be marked with standard start and end markers.

10. The log format should be structured and delimited such that it is reasonably human readable (such as tab delimited).

11. The level of detail included in the log file should be configurable in terms of preset levels: 1- error and service events only, 2- Functional events, error events, service events, 3- All events including diagnostic information messages about internal states and parameters.

12. The log should behave as a ring buffer with a maximum of X events (where X is configurable within some limit that would not be exceeded in 7 days of heaviest anticipated use). If the size of the log exceeds available space, the oldest events shall be discarded first.

13. When the log is exported, it should have a header that identifies the software version (and serial number of the HW, if applicable) and current configuration.

Could the Military Be Waking Up?

Ever since I got into the testing field, almost 20 years ago, it’s been a truism that military software development is moribund. It’s not that they love process, it’s that they love bad process. Documentation? Bad documentation. Who can look upon 2167A or Mil-Std-499 without dismay? I’ll tell you who: paper mills and people paid to arrange the ink on all that paper. It’s just a scam and a travesty.

I was asked, in 1998, to analyze two military software software test plans, each for a different major weapons system. I told them up front that I was neither interested nor qualified to assess the test plans against DoD standards, such as Mil-Std-499. I was told, no problem, assess them against best commercial practice. Interpreting “best commercial practice” as what I would generally recommend doing for a life-critical or otherwise high stakes project in the commercial sector, I created a model for analyzing test plans (now published on my website and also in the appendices of Lessons Learned in Software Testing). I then applied the model to the test plans I was given. What was immediately apparent is that the military test documentation had very little information density. It looked like the 75-page documents had been automatically generated by a set of macros operating on a short bullet list.

I made a bunch of suggestions, including showing how the same information could productively be packaged in 5 pages or so. That way, at least we could wage war on something other than trees. They replied, after two months of silence, that my analysis was not useful to them. Why? Because my ideas about test plan documentation were not consistent with Mil-Std-499. That was the only feedback I received about that work. Way to squander taxpayer money, guys! Hoowaa!

A New Hope

The defense department may be waking up to the problem, at long last. See the NDIA Top Issues Report. Notice that two issues I like to harp about are in the top five challenges: skills and testing. Notice that the military is now officially concerned about wasteful documentation and low-skilled workers.

Maybe not coincidentally, I recently taught my class at Eglin Air Force base, with F-15s thundering regularly overhead. I was surprised that they would invite me to teach there. I was a bit more surprised that they were quite receptive to the concept of skilled and agile software testing, wherein our documentation is produce only because and only to the extent that it actually serves a useful purpose.

Studying Jeff Atwood’s Paint Can

I just found Jeff Atwood’s Coding Horror blog. He’s an interesting writer and thinker.

One of his postings presents a good example of the subtle role of skill even in highly scripted activities. He writes about following the instructions on a paint can. His article links to an earlier article, so you might want to read both.

The article is based on a rant by Steve McConnell in his book Rapid Development about the importance of following instructions. Steve talks about how important it is to follow instructions on a paint can when you are painting.

I want to talk, here, about the danger of following instructions, and more specifically, the danger of following people who tell you to follow instructions when they are not taking responsibility for the quality of your work. The instruction-following myth is one of those cancers on our craft, like certification and best practices.

[Full Discolosure: Relations between me and McConnell are strained. In the same book, Rapid Development, in the Classic Mistakes section, Steve misrepresented my work with regard to the role of heroism in software projects. He cited an article I wrote as if it was indicative of a point of view that I do not hold. It was as if he hadn’t read the article he cited, but only looked at the title. When I brought the error to his attention, he insisted that he did indeed understand my article and that his citation was correct.]

Let’s step through some of what Jeff writes:

“But what would happen if I didn’t follow the instructions on the paint can? Here’s a list of common interior painting mistakes:

The single most common mistake in any project is failure to read and follow manufacturer’s instructions for tools and materials being used.”

  • Jeff appears to be citing a study of some kind. What is this study? Is it trustworthy? Is Jeff himself telling me something, or is Jeff channelling a discarnate entity?
  • When he says “the most common mistake” does he mean the one that most frequently is committed by everyone who uses paint? Novices? Professionals? Or is he referring to the most serious mistakes? Or is he referring to the complete set of possible mistakes that are worth mentioning?
  • Is it important for everyone to follow the instructions, or are the instructions there for unskilled people only?
  • Why is it a “mistake” not to read-and-follow instructions? Mistake is a loaded term; one of those danger words that I circle in red pencil and put a question mark next to. It may be a mistake not to follow certain instructions in a certain context. On the other hand, it may be a mistake to follow them.

Consider all the instructions you encounter and do not read. Consider the software you install without reading the “quickstart” guide. Consider the clickwrap licenses you don’t read, or the rental cars you drive without ever consulting the drivers manual in states where you have not studied the local driving laws. Consider the doors marked push that you pull upon. Consider the shampoo bottle that says “wash, rinse, repeat.” Well, I have news for the people who make Prell: I don’t repeat. Did you hear me? I don’t repeat.

I would have to say that most instructions I come across are unimportant and some are harmful. Most instructions I get about software development process, I would say, would be harmful if I believed them and followed them. Most software process instructions I encounter are fairy tales, both in the sense of being made up and in the sense of being cartoonish. Some things that look like instructions, such as “do not try this at home” or “take out the safety card and follow along,” are not properly instructions at all, they are really just ritual phrases uttered to dispel the evil spirits of legal liability. Other things that really are instructions are too vague to follow, such as “use common sense” or “be creative” or “follow the instructions.”

There are, of course, instructions I could cite that have been helpful to me. I saw a sign over a copy room that said “Do not use three hole paper in this copy machine… unless you want it to jam.” and one next to it that said “Do not use the Microwave oven while making copies… unless you want the fuse to blow.” I often find instructions useful when putting furniture together; and I find signs at airports generally useful, even though I have occasionally been steered wrong.

Instructions can be useful, or useless, or something in between. Therefore, I propose that we each develop a skill: the skill of knowing when, where, why and how to follow instructions in specific contexts. Also, let’s develop the skill of giving instructions.

Jeff goes on to write:

“In regard to painting, the most common mistakes are:

* Not preparing a clean, sanded, and primed (if needed) surface.
* Failure to mix the paints properly.
* Applying too much paint to the applicator.
* Using water-logged applicators.
* Not solving dampness problems in the walls or ceilings.
* Not roughing up enamel paint before painting over it.”

Again with the “most common.” Says who? I can’t believe that the DuPont company is hiding in the bushes watching everybody use paint. How do they know what the most common mistakes are?

My colleague Michael Bolton suggested that the most common mistake is “getting the paint on the wrong things.” Personally, I suspect that the truly most common mistake is to try to paint something, but you won’t see THAT on the side of a paint can. As I write this, my bathroom is being repainted. Notice that I am writing and someone else is painting. Someone, I bet, who knows more about painting than I do. I have not committed the mistake of trying to paint my own bathroom, nor of attempting to read paint can instructions. Can I prove that is the most common mistake? Of course not. But notice that the rhetoric of following instructions is different if you draw a different set of lines around the cost/value equation represented by the word “mistake.”

Also, not knowing much about painting, I don’t understand these “mistakes.” For instance:

  • What is a clean surface? How do I sand it? What does “primed” mean and how do I know if that is needed?
  • How do I mix paints? Why would I even need to mix them? What paints should I mix?
  • What is the applicator and how do I apply paint to it? How much is enough?
  • What is a “water-logged” applicator? How does it get water-logged? Is there a “just enough” level of water?
  • How does one recognize and solve a “dampness problem”?
  • I assume that “roughing up” enamel paint means something other than trying to intimidate it. I assume it means sanding it somehow? Am I right? If so, how rough does it need to be and how do I recognize the appropriate level of roughness?

I am not kidding, I really don’t know this stuff.

Then Jeff writes:

“What I find particularly interesting is that none of the mistakes on this checklist have anything to do with my skill as a painter.”

I think what Jeff meant to say is that they have nothing to do with what he recognizes as his skill as a painter. I would recognize these mistakes, assuming for the moment that they are mistakes, as being strongly related to his painting skill. Perhaps since I don’t have any painting skill, it’s easier for me to see it than for him. Or maybe he means something different by the idea of skill than I do. (I think skill is an improvable ability to do something) Either way, there’s nothing slam dunk obvious about his point. I don’t see how it can be just a matter of “read the instructions stupid.”

Jeff writes:

“My technical proficiency (or lack thereof) as a painter doesn’t even register!”

Wait a minute Jeff, think about this. What does have to do with your proficiency as a painter? You must have something in mind. If proficiency is a meaningful idea, then you must believe there is a detectable difference between having proficiency and not having proficiency, and it must go beyond this list of issues. Rather than concluding that your skill doesn’t enter into it, perhaps one could look at the same list of issues and interpret it as a list of things unskilled people frequently do when they try to paint things that often leads them to regret the results. It’s a warning for the unskilled, not a message for skilled painters. A skilled painter might actually want to do these things, for instance, to paint with a water-logged applicator to get some particular artistic effect.

Jeff writes:

“To guarantee a reasonable level of quality, you don’t have to spend weeks practicing your painting skills. You don’t even have to be a good painter. All you have to do is follow the instructions on the paint can!”

Now I have logic vertigo. How did we get from avoiding obvious mistakes, where we started, to “reasonable quality”? Would a professional house painter agree that there is no skill required to achieve reasonable quality? Would a professional artist say that?

(And what is reasonable quality?)

Even following simple instructions requires skill and tradeoff decisions. A paint can is neither a supervisor, nor a mentor, nor a judge of quality. Don’t follow instructions, instead I suggest, consider applying this heuristic: instructions might help.

And one more thing… Does anyone else find it ironic that Jeff’s article about reading instructions on paint cans would include a photo of a paint can where the instructions have been partly obscured by paint? Perhaps the first instruction should be “Check that you see this sentence. If not, please wait for legible instructions.”

Fighting Bad Test Documentation

A lot of people I teach seem to be under pressure to create more documents to describe their test process. But documenting is not testing. It is one of the chief distractions to testing.

“James Bach hates documentation!”, some people will say. No, I don’t. I hate waste. I hate boring clerical work that unnecessarily interrupts productive and interesting intellectual work. You should hate it too.

I’m a writer for cryin’ out loud. I like documentation– when it is the solution to a problem that matters, and not merely a management obsession. But if you’re trying to move to concise test documents (outlines, matrices, one-page reference sheets, and other minimalist formats) you may need help persuading management and co-workers.

Here are some ideas:

  • Show management how much less testing we are able to do because we are spending so much time with documents.
  • Show management how certain kinds of testing isn’t done at all just because it is hard to document (exploratory testing and complex scenario tests often fall in this category). This is perhaps the most chilling effect of over-documentation, especially in the realm of medical devices. I keep seeing medical device test doc that is simplistic, in all its obesity, to the point of near worthlessness.
  • Examine closely what testers are doing and show that they aren’t even following the documentation (often they aren’t, in my experience as a consultant who audits test processes).
  • Demonstrate the power of exploratory testing (a less heavily documented approach). One day of ET is often sufficient to find what would take a week to find when following detailed documented test procedures.
  • Demonstrate the value of concise test documentation (matrices, outlines).
  • Consider documenting at the level of test activities rather than test cases.
  • Consider automatic documentation (via log files produced by the product under test or via an external logging tool such as Spector).
  • Ask the question: what exactly are we getting from our documentation? Don’t accept any theoretical answers. For example, one typical answer is that documentation protects the company from the ill effects of staff turnover. But does it? Probably not, in my experience. That’s a theory based on ignorance about how people learn. In real life, new testers learn what they need to know by playing with the product itself, and talking to the people around them. In my experience, testers come up to speed in a few days at most. And in my experience, test documentation is often of such poor quality that it’s better ignored than followed. You have to go by your own experience, of course. I’m just suggesting that you ask the questions and make your own observations.

Here are a few books about documentation that might help you make your case:

Heavy documentation is often a consequence of managers and testers who just aren’t thinking through the reasons why they do things. They hear that documentation is good, but they don’t stop to consider the cost of documentation, or watch how documentation is actually used (or more often, ignored).