[James’ Reply: By my definitions, the opposite is the case.]

I couldn’t think of a model that wasn’t a ‘mechanism by which one would recognize a problem.’ (definition of an Oracle)

[James’ Reply: You might as well tell me that you can’t think of an object that is not a “chair”, because, in principle, any object could be used as a chair. Perhaps any model COULD be used as an oracle. So what? The interesting question is what oracles ARE you using in each specific circumstance.]

But in reviewing James’ mnemonic for Oracles (HICCUPP), there are plenty of non-models: History, Image, Comparable Product, etc. Though I suppose one could argue that once an Oracle is invoked, it creates a model of expected behavior and thus we’re back to Model=Oracle and Oracle=Model.

[James’ Reply: On the contrary, all of those involve models. History is a model (our notion of history is a biased and idiosyncratic selection and construction of what once was true or what transpired before, and mental models control that selection/construction), and image (looking professional requires a model of what “professional” is), and certainly comparable products (where one product is used as a model for another!). What do you think a model is?]

Apologies if I’ve dived into nit-picky semantics. Maybe it’s just me, but I find the interchangability of these two words to be very confusing. My brain ‘trips’ on them all the time.

[James’ Reply: Nitpicking is fine, in this case. It helps learning. The two words are certainly not interchangeable. Perhaps you feel they are because you are looking for essences instead of usage. Anything can be a model. Anything can be an oracle. But that depends not on its essence, but rather its role in our thoughts. Imagine a banana. That can be a model, if I use that banana to help design a banana-shaped opera house. If I also (or instead) use it to check the correctness of the shape of supposedly banana-shaped opera house, it becomes an oracle as well as a model. Were I to stop using it in those ways, it goes back to being just a banana. Sometimes a banana is just a cigar, after all.]

Similarly, all oracles are heuristic, but not all heuristics are oracles.]

This was sparked by Ben’s comment above, where he says: “I also simplify the MODEL definition by defining some things as data and others as states. In your example, I would likely make the clock a data item (rather than a different state) and add ORACLES to my test to confirm that the clock is ticking in the expected direction.”

In this case, it sounds like Ben is adjusting his “Model” to have certain expecations of the software, and if they don’t match then he recognizes a problem (ergo, “Oracle”).

[James’ Reply: You can use any sort of model as an oracle. The mark of an oracle is how it helps you recognize a problem. The mark of a model is that it is an idea, activity, or object that represents another idea, activity, or object, such that understanding or manipulating the model helps you understand or manipulate the thing that it represents.]

Interesting presentation. I would make the a couple of observations.

Your presentation seems to make a better case for exploratory testing than any form of model-based testing. However I would argue that you were not testing the app so much as learning its observed behavior and then itemizing what seemed “unusual.” If the point of the app was a quick and dirty promotion to sell high speed connectivity, then most of your list of behaviors would probably not be classified as bugs.

[James’ Reply: Well, that’s not really the point. I don’t care, in this case, what someone would consider a bug. My point is that any application, even a simple one, is more complex than we can conveniently model in any formal way. And I’m arguing that we must test in a way that admits surprises and helps us expand our models in real-time.] 

Exploratory understanding of the app is much better done against the specification, when the developer can benefit rather than after the code is written and rework becomes expensive. Exploratory testing remains a terribly unmanageable in inefficient process for testing that an application conforms to specification.

[James’ Reply: I don’t know where your assertion is coming from. What experience do you have with exploratory testing? I suspect you may be using a different definition of it than we do who foster this approach. You’re calling it unmanageable… Well, do you mean you don’t know how to manage it? Or are you saying that no one can manage it? If the latter, I find that surprising, considering that I can manage it, I do manage it, and I teach other people how to manage it.] 

I would challenge the notion of being sceptical of model-based testing. Rather, the issue is to understand thoroughly the types of coverage which each model approach provides and apply the appropriate ones to the testing problem.

[James’ Reply: If you want to challenge, then go ahead and challenge, but where’s your challenge? What I see is a counter-assertion without any reason provide. Please let me hear your argument. Skepticism is a powerful attitude to have in testing. I demonstrated specifically, in my talk, why any model– any model at all– is going to limit you. This is not necessarily a problem, but it is a problem if you fall in love with a particular model, or with the idea of testing exclusively against formal models. It is my skepticism that protects me from such obsessions.] 

For example, 70% of defects are the result of incorrect implemnetation of required functionality. Only 15% of bugs are actual coding errors. Cause-effect models provide test cases with very thorough functional coverage. So a C-E model is preferable over a code structure model for catching the most prevalent bugs. State models exercise state transitions but may not exercise all of the conditionis which result in a state change. So if I am testing a cell phone app, a blend of C-E and state models are required to get adquate coverage.

[James’ Reply: Where did you get these numbers? Without context, they have no meaning or value to me. You might as well say “somebody somewhere at some time measured the temperature of something and the temperature was 70 degrees Fahrenheit, therefore the temperature where you are must be…” Come on, man. Don’t use numbers that way. Anyway, I would be happy to evaluate your argument once you put it in some sort of meaningful form. There are so many suppressed premises and assumptions that I can’t make sense of it. You can start by defining some of your terms. What do you think “functional testing” means? What do you think a cause-effect model is? Then show me a “cause-effect model” and tell me why I should believe that it represents the application that you claim it represents? Peter, I deal in fabulous and overwhelming complexities. I am a tester. I’m not a model believer, I’m a model questioner. I AM A TESTER.] 

The value of formal model-based testing is that it provides a consistent and reproducable approach to designing test cases and obtaining measurable test coverage. That means manageability of the test process and predictability of time and cost. In our testing events, we pick the modeling approaches appropriate to the problem. This allows us to meet time and budget constraints, manage and track the process and end up with very high reliability applications delivery consistently.

Pete Becker
Critical Logic

[James’ Reply: Peter, I would urge you to be more self-critical. I think you have achieved the illusion of manageability, and you have achieved this illusion by avoiding deep questions about your models and their intellectual and empirical foundations. I don’t offer my clients formulas for success. I offer them a way to gain the necessary skills to avoid fooling themselves.]

I am a skeptic and agree that a healthy amount of skepticism is necessary to be a good tester. I apply my “defensive pessimism” to testing. (See the book “The Positive Power of Negative Thinking” or http://www.defensivepessimism.com.) Although always considering what I (and others) might be missing and what might go wrong probably isn’t a good way to win friends, it does contribute to good testing.

I agree with your point: ALL testing is model based. Mental models are the basis for all testing. Scripted testing is based on a testers mental model at the time of scripting; which is why I don’t like scripted automation: it is a specific implementation of part of an explicit model. An engaged manual tester updates their mental model as they learn during testing.

I am certain my automation is missing important bugs that could be found by “interactive, attentive, sentient, cognitively intensive testing”. Models are simplifications of the real thing, as automated testing is a simplification of the real thing. I find that automating tests with explicit models can be (if the automation designer thinks of the right things) less of a simplification than automating scripted test procedures.

My models are full of assumptions and (as you point out) my software cannot question those assumptions but it can report anything it finds that deviates from my assumptions. I can then analyze the reported data (another task that takes an engaged human mind) to refine my mental model. In this way, automation can be a great testing tool.

I think that the idea of scripting test cases came from a time that developers wrote code with pencil and paper, punched it into cards, waited for computer time, and finally attempted to compile the code. Coding software is now a more interactive activity. Developers can try things and instantly compile and test their ideas. For some reason, many testing practices are still based on the way software was developed 40 years ago.

[James’ Reply: Thanks for commenting, Ben. This makes sense to me.]

Automated model-based testing does not need to be as light as you suggest. You have appropriately identified the limitations that prevent automated model-based testing from scaling beyond simple application components.

I’ve been implementing automated model-based tests since I read Harry’s article in the September 2000 STQE magazine (http://www.geocities.com/model_based_testing/intelligent.pdf).

I believe that most automated UI testing should be model-based. And I am a proponent of exploratory manual testing. As someone that makes a living from automated testing, I often find myself in the awkward position of trying to communicate the value of manual testing when asked to automate “everything”. I want my test automation to provide value to the manual testers – not attempt to replace them. I am a firm believer in your Rule #1 of test automation: “A good manual test cannot be automated.”

I don’t like most purely scripted testing — whether it be manual or automated. Testing needs to be an interactive activity – whether it be manual or automated.

Test automation should be a tool to assist the manual tester – just as most software assists human users instead of replacing them. The industry needs to stop treating test automation as a batch process that runs over and over again without human intervention. I believe that many automation efforts fail because they are attempts to replace the manual tester.

Automated model-based testing allows me to simplify the automated test creation and maintenance. As with any test automation, the automation is only as good as what the designer put into it. If unexpected input is not modeled, it will not be tested. And a big part of any automation maintenance is to enhance the model based on new knowledge and ideas.

I use automated model-based testing to test whole applications. (Not to imply that I wholly test any application.) I am not manually defining every possible state transition. Most modern software applications are hierarchical in nature. I address the state explosion problem by using hierarchical state machines. In your example, I would only need to define the right-click actions once and then let the computer flatten the model into all the possible transitions.

[James’ Reply: The basic point I’m arguing for is that ALL testing is model-based and ANY explicit model is a simplification that may be an oversimplification. So, from my point of view, while I agree that you can just right-click to add the actions, the problem is that you have to think of doing that, and you have to decide whether it’s worth doing that. This is not a trivial matter. The unbearable lightness is the trouble we have getting to a model that we ought to trust; that is justifiable and sufficient.] 

I also simplify the model definition by defining some things as data and others as states. In your example, I would likely make the clock a data item (rather than a different state) and add oracles to my test to confirm that the clock is ticking in the expected direction.

[James’ Reply: You would do that if you thought of it, and thought that it was worth doing.] 

You say that I am not going to make a 1400 box state model. You are correct. I let the computer make it for me from one or more simple hierarchical state models. I am running automated model-based tests with hundreds of thousands of state transitions — and the automation is finding important bugs missed by manual testing.

[James’ Reply: Your automation is probably also missing important bugs that could be found by interactive,  attentive, sentient, cognitively intensive testing. This is because your model is a simplification. The computer cannot, in fact, “make it for you”. What it’s doing is fulfilling an algorithm it has been given, and that algorithm is full of assumptions that cannot be questioned by your software. The computer is expanding a model that you have already provided.]

Hierarchical state machines even let me easily validate my assumptions about the behavior of an application. For example, if I expect the same behavior from pressing a button no matter what the state within an application, I can define the behavior once and let the computer run by itself and report any states from which the behavior is different. This information can then be used to help direct manual testing and future improvements to the model.

[James’ Reply: I think the point you are making is that model-based test tools can be wonderfully helpful. I agree. That’s why I use them. However, I use them (and I suspect you do, too) with a healthy skepticism and ongoing vigilance for things that may be missed.]

[James’ Reply: It’s a private forum where I encourage readers to attempt to post their questions.]

[James’ Reply: Well, maybe not unimaginable.]