Heuristic Value is a Relationship

One of the comments on my post about The Esssence of Heuristics went like this:

“An excellent example of a heuristic is a hammer.”


Ecstasy is your friend: it picks you up at the airport.

Non heuristics that can help an expert solve a problem, without being a guarantee – an abridged list:
* Money
* Time
* Expertise
* Newton-Raphson Algorithm
* Analogies

It was posted anonymously. I generally don’t publish anonymous comments that are argumentative, because opponents who yell things and run away bore me… but this one is helpful. The writer is a little confused, but I bet other people are confused too, in the same way.

He offers a list of things that he claims are non-heuristics (he doesn’t explain why so I guess he thinks they are “obviously” non-heuristics), and suggests that they meet my definition. I think he’s trying to make two points: either my definition of heuristics is wrong because it admits non-heuristics, or it’s trivial because then everything would be a heuristic and therefore the idea conveys no information.

Well, the first point is easily handled. By definition, each thing on his list is heuristic (because he has declared that they help without guaranteeing in some scenario which he seems to have in mind). There is no contradiction, he’s simply mistaken by calling those things non-heuristics.

As to his second point, that’s where the real confusion lies. I think he is mixed-up because he expects heuristics to have some essential nature that identifies them. But what makes something a heuristic is not its essential nature, but rather its relationship to the problem and the problem-solver. We say that anything that may help you solve a problem has “heuristic value” for you with respect to that problem. But if it is infallible (and also if it halts) then we don’t call it a heuristic. We call it an algorithm. For instance, long division is a guaranteed way for dividing one rational number with a finite number of digits by another such number. However, long division is just a heuristic if we were to apply it to splitting the check at a restaurant. It doesn’t guarantee that everyone will feel that they are paying a fair share.

How about instead of heuristics we think about weapons? Would it be absurd of me to suggest that a weapon is any fallible method of attacking your enemy or winning a battle? Someone might reply “But James, are pickles really weapons? Sand? Wind?” The answer is: they can be, depending on your situation. But you don’t need to catalog all possible weapons. Instead you study the art of combat and learn to spot the things that will help you. Myamoto Musashi wrote exactly about this in 1645:

When you attack the enemy, your spirit must go to the extent of pulling the stakes out of a wall and using them as spears and halberds. — Book of Five Rings

We cannot speak intelligibly about heuristics without identifying the problem, the problem-solver, and the dynamics of the situation. You can make a list of anything you want, Mr. Anonymous, but can you answer my next questions, which are: what specific problems are you talking about for which these are heuristics and how specifically are they heuristic? Or if you think they aren’t heuristics, what specific problem do you think they can’t help with and why not?

Three New Testing Heuristics

A lot of what I do is give names to testing behaviors and patterns that have been around a long time but that people are not systematically studying or using. I’m not seeking to create a standard language, but simply by applying some kind of terminology, I want to make these patterns easier to apply and to study.

This is a quick note about three testing heuristics I named this week:

Steeplechase Heuristic (of exploratory boundary testing)

When you are exploring boundaries, think of your data as having to get to the boundary and then having to go other places down the line. Picture it as one big obstacle course with the boundary you are testing right in the middle.

Then consider that very large, long, extreme data that the boundary is designed to stop might founder on some obstacle before it ever gets to the boundary you want to test. In other words, a limit of 1,000 characters on a field might work fine unless you paste 1,000,000 characters in, in which case it may crash the program instantly before the boundary check ever gets a chance to reject the data.

But also look downstream, and consider that extreme data which barely gets by your boundary may get mangled on another boundary down the road. So don’t just stop testing when you see one boundary is handled properly. Take that data all around to the other functions that process it.

Galumphing (style of test execution)

Galumphing means doing something in a deliberately over-elaborate way. I’ve been doing this for a long time in my test execution. I add lots of unnecessary but inert actions that are inexpensive and shouldn’t (in theory) affect the test outcome. The idea is that sometimes– surprise!– they do affect it, and I get a free bug out of it.

An example is how I frequently click on background areas of windows while moving my mouse pointer to the button I intend to push. Clicking on blank space shouldn’t matter, right? Doesn’t hurt, right?

I actually learned the term from the book “Free Play” by Stephen Nachmanovitch, who pointed out that it is justified by the Law of Requisite Variety. But I didn’t connect it with my test execution practice until jogged by a student in my recent Sydney testing class, Ted Morris Dawson.

Creep & Leap (for pattern investigation)

If you think you understand the pattern of how a function works, try performing some tests that just barely violate that pattern (expecting an error or some different behavior), and try some tests that boldly take that behavior to an extreme without violating it. The former I call creeping; the latter is leaping.

The point here is that we are likely to learn a little more from a mildly violating test than from a hugely violating test because the mildly violating test is much more likely to surprise us, and the surprise will be easier to sort out.

Meanwhile, stretching legal input and expectations as far as they can reasonably go also can teach us a lot.

Creep & Leap is useful for investigating boundaries, of course, but works in situations without classic boundaries, too, such as when we creep by trying a different type of data in a function that is supposed to be rejected.

The Essence of Heuristics

Excellent testing requires skill, but heuristics give structure to that skill. Heuristics help us access our skills under pressure.

A heuristic is a fallible method of solving a problem or making a decision. Cem Kaner and I came to this definition based on an extensive search of papers and books across fifty years of psychology and engineering. Amazingly, we were not able to find a single coherent definition of heuristic in our research (The dictionaries are not much help, here) . Coleridge, Kant, and Polya have all written about heuristic reasoning. But we needed a definition that would bring the issues into focus.

There are two main issues: something that helps you solve a problem without being a guarantee. This immediately suggests the next issue: that heuristics must be applied sapiently (meaning with skill and care).

An excellent example of a heuristic is a hammer. Do you see how a hammer can help a carpenter solve a problem, but does not itself guarantee the solution? I like this example because it’s so easy to see that that a hammer may be critical to a skilled carpenter while being of little use to an unskilled lout who doesn’t know what to pound or how hard to pound it or when to stop pounding.

Heuristics do not replace skill. They don’t make skill unnecessary. But they can make skilled people more productive and reliable.

How Do You Tell if Consultants/Trainers Understand Heuristics?

I typically hear two reactions from rival testing consultants of other schools of thought (especially the Factory and Quality Control schools) that love “best practice” talk:

  1. “Oh yes, heuristics! That’s another name for ‘best practices’, right?”
  2. “Oh no, heuristics! That’s just a fancy name for ‘best practices’!”

Obviously both reactions miss the point. Even if these folks rename their ideas of what you should do to call them “heuristics,” they would be leaving out a key idea, which is that skill must rule methods. Talking about methods, focusing on methods, enshrining methods, is only sensible if humans are left in charge. The heuristic nature of engineering is the reason why a “best practice” is an absurdity. Seek not the perfect practice. Seek instead to practice your skills.

My friend Judah Mogilensky once commented that he thought my Heuristic Test Strategy Model was a lot like the Capability Maturity Model that he works with.

“Are people allowed to change or suspend the CMM whenever they see fit?” I asked.

“Oh no,” he replied.

“Then it isn’t being applied responsibly as a heuristic. It’s being treated like a law, or an oath, or something else that places itself above its subjects and binds them.”

So that’s how you tell. Look to see what’s driving things. People, or concepts? Fundamentally, “methodology” can’t control projects. Anytime you seem to see methods in charge, you are actually witnessing a project out of control, or else a project under covert control by shadowy masters.

When someone teaches you a way to solve a problem, check:

  • Do they teach you how to tell if it’s working?
  • Do they teach you how to tell if it’s going wrong?
  • Do they teach you heuristics for stopping?
  • Do they teach you heuristics for knowing when to apply it?
  • Do they compare it to alternative heuristics?
  • Do they show you why it works?
  • Do they help you understand when it probably works best?
  • Do they help you know how to re-design it, if needed?
  • Do they let you own it?
  • Do they ask you to practice it?
  • Do they tell stories about how it has failed?
  • Do they listen to you when you question or challenge it?
  • Do they praise you for questioning and challenging it?

Stuart Reid’s Bizarre Plea

Stuart Reid is planning to do a talk on how we should use “evidence” in our debates about what works and doesn’t work in testing.

A funny thing about that is Stuart once spent 30 minutes trying to convince me that the number “35,000” was evidence of how great the ISEB certification is, as in “35,000 happy customers can’t be wrong.” Such a concept of “evidence” wouldn’t not pass muster in a freshman course in logic and rhetoric. How does he know that the 35,000 people are happy? How does he know that they are qualified to judge the quality of the certification? How does he explain the easily checked fact that you can pick out any three ISEB or ISTQB certified testers, ask them if they think the certification has made them better testers or indicates that they are better testers, and at least two of them will do the equivalent of rolling their eyes and smirking? (Don’t believe me? I understand. So TRY IT, as I do on a regular basis in my classes)

You might think Stuart is attempting a bold and classic rhetorical move: attempting to control the terms of the debate. The problem he has is that he will lose the debate even faster if he actually engages on the question of evidence. This is because there is plenty of evidence from other fields and the history of thought itself to justify the positions of the Context-Driven School of testing. We are winning the debates because we are better informed and better educated than the Factory Schoolers, for instance, represented by Reid. For instance, Rikard Edgren (who says he’s not in the Context-Driven School, but looks like a duck to me) wrote about applying Grounded Theory to testing. I wonder if Stuart Reid has ever heard of Grounded Theory. He probably has, because I probably mentioned it at least once in the hours of debate that Stuart and I have had. He didn’t respond or react. My impression was that he wasn’t listening.

There’s something far more important than evidence that we need in our industry: engagement. People need to listen to and respond to the arguments and evidence that are already out there.

Here’s one sort of evidence I put in front of Stuart, in a debate. I claimed that my school of testing represents a different paradigm of thinking about testing than his does. After giving him examples of specific words that we define differently and concepts that we arrange differently, it became clear that the deeper problem is that he thought I was pretending to believe things that I don’t believe, just to be difficult. He actually said that to me!

This is the last resort of the determined idealogue: poke your own eyes out so that you don’t risk seeing contrary evidence. Stuart’s case rests on pretending that no one else is making a case! His demand for evidence is meant to give the impression that the evidence is not already sitting in front of him being ignored.

Cem Kaner, Michael Bolton, and I have been marshaling evidence, pointing out the lack of evidence against our ideas, and demonstrating our methods for many years. Next week it will be exactly 23 years since I first became a full-time software tester, and nearly 17 years since the first time I stood up at a conference and pointed out the absurdity of “traditional” testing methods.

BTW, here some of the kinds of evidence I offer when challenged about my work:

  • The Sciences of the Artificial, by Herbert Simon (this establishes, based on a body of research for which he won the Nobel Prize in 1978, the heuristic nature of engineering)
  • Collaborative Discovery in a Scientific Domain, Takeshi Okada, Herbert Simon, 1997, (this is an experiment that observed the behaviors of scientists attempting to create and perform experiments together in an exploratory way)
  • The Processes of Scientific Discovery: The Strategy of Experimentation, Deepak Kulkarni, Herbert Simon, 1988 (this study analyzes the basic exploratory processes of science)

The first item here is a book, the next two are papers published in the journal Cognitive Science. See, if Stuart wants evidence, he has to look beyond the desert that is Computer Science. He needs to get serious about his scholarship. That will require him to find, in his heart, a passion to learn about testing.