This post was a criticism of a post by Paul Gerrard that has been withdrawn. Hence, I withdraw my criticism.
Recently, Michael Bolton and I had a conversation, which he recorded, about the deep nature and meaning of a scripted test procedure. It’s an MP3 file that is a bit more than an hour long.
If you think that scripting a test is a simple matter of “writing down what the tester should do”, then I would urge you to listen to Michael and I deconstruct a simple example of a test script.
We take the first step of a hypothetical test and analyze it. That step is:
“1. Reboot the test system.”
What does this mean? What should the tester do? Consider these questions for a moment before listening to the MP3.
It would be an unsound fancy and self-contradictory to expect that things which have never yet been done can be done except by means which have never yet been tried.
This reminds me of that “definition of insanity” which is so often attributed to Einstein. But this comes from Francis Bacon, circa 1620. In his seminal work “The New Organon”.
Bacon has some other interesting quotes for testers…
The logic now in use serves rather to fix and give stability to the errors which have their foundation in commonly received notions than to help the search after truth. So it does more harm than good.
This is how I feel about the ISTQB syllabus.
There is no soundness in our notions, whether logical or physical. Substance, Quality, Action, Passion, Essence itself, are not sound notions; much less are Heavy, Light, Dense, Rare, Moist, Dry, Generation, Corruption, Attraction, Repulsion, Element, Matter, Form, and the like; but all are fantastical and ill defined.
This is how I feel about a lot of testing terminology.
It is idle to expect any great advancement in science from the superinducing and engrafting of new things upon old. We must begin anew from the very foundations, unless we would revolve forever in a circle with mean and contemptible progress.
I feel that way, too, about patterns such as the v-model, and most of what passed for test techniques in the 80’s.
Francis Bacon was proposing a great break with the stifling Aristotelianism of his day (the certification craze of the middle ages), and sought a new foundation for science. Bacon thereafter became a godfather of the enlightenment, helping to create the modern world.
We’re way beyond Bacon, now. Still, I’m attracted to his sentiment that what passed for good scientific work in his time was actually nothing but uncritical folklore. In our time, in our little field, we need a similar re-invention of the craft, a New Organon of testing.
People have been asking me why I haven’t been blogging. Well, here it is. I arrived in London, today. I’m going to Scotland next week. I just came from Sweden. I arrived in Sweden from Australia, by way of New Zealand. This is the longest and most intense road trip I’ve done since I started my company, eight years ago. I’m tired! No, here’s a better word: I feel dazed. If I’m not traveling, then I’m talking. My life for the last five weeks has been one conversation and demonstration after another, every day, even weekends.
I’ve taught at specific companies as well as doing public classes. I did a conference in Sweden and a SIG meeting in Christchurch. I consulted for two medical device companies that are using exploratory testing, and did a rapid testing clinic for an electronics company. I’m hoping they will let me write about what happened there. It was wonderful for me to see people demonstrating Rapid Testing on the job.
Although I have not been blogging, I have been getting lots of ideas. So much to talk about: new heuristics, the “mipping” of bugs, the importance of dynamic focus, latent testing skills, how to hold a testing clinic, doing standing state testing, verbal self-defense for testers, and my portable test coaching kit. Lots of stuff.
Adelaide and Christchurch were wonderful as usual. But I’m really coming to love Sweden and the Swedish testing spirit. To be sure, I’ve only seen a bit of it. But that bit is impressive. I make a point of meeting with Tobbe Ryber each time I visit. He taught me a game we call “plenty questions” that I will be blogging about, at some point. Pablo Garcia charmed me with wine, cheese, and his cheerfully incisive outlook on the testing world. Anders Claesson revealed himself to be a philosophical soul. Imagine someone with the voice and look of an accountant, but the subtle mind and resources of Gandalf the Grey– if Gandalf had been a tester.
Context-driven testing has been a predominantly American community. Some of it is also happening in England. But Sweden appears to be the emerging hotbed. Someone told me that the Swedish management culture is fond of diversity and individual initiative combined with teamwork. Maybe that explains it.
Michael Hunter interviewed me for DDJ. See the interview here.
Michael’s a cool guy. He just took the Rapid Software Testing class. It’s exciting to me when a Microsoftie takes the class, considering how hostile Microsoft has become to the idea that testing could be a skilled activity, distinct from programming.
I’ve taught two segments of the Rapid Software Testing Online class, so far. For me, it’s working better than I expected. Here are some of the things I’ve learned about online training, so far:
A Surprise
The most surprising thing to me about the class is how much MORE involved some of the students are, intellectually, than when I do the class as a three-day event. I think this is a side-effect of splitting the class into pieces and doing them over time, with the Moodle forum providing an ongoing focus for processing what happened.
I’m now feeling pretty enthusiastic about commercializing the online version of the class. It’s more work than the in-person version, but it’s also, I believe, a more valuable learning experience for being stretched over time and allowing more of a dialog.
If you are interested in taking Rapid Software Testing Online, I will start one once I have 15 paying customers. The cost will be $249 per session. Students pay on a session-by-session basis, but each session is a prerequisite for the next. The fee includes access to the class forum.
The course will consist of six to nine 3-hour sessions, starting at 11am EST, scheduled as close to a week apart as I can. I’m working on a new class outline and a schedule to lay all this out.
If you are interested, email me at james@satisfice.com.
Last night I gave a webinar to the CAMUG people in Calgary. About forty people attended online, as well. You can see this presentation on Google Video. The resolution is poor, so you may also want to see the slides.
The talk itself is 20 minutes long, then I answer questions for an hour.
In this talk, I tried to illustrate why any testing based on a model is inherently limited, and why we must strive against those limitations. I talk about some ways of doing that.
Numerous quesitions came up during the talk. I answered many. Below are the questions that were typed to me that I did not have a chance to answer, during the event:
How often do we apply blink testing?
I apply blink testing any time I can arrange to be confronted with a blizzard of data: comparing screens (I glance at a million pixels, then at another million pixels, and in an instant I see the tiny difference between them), scrolling through huge log files, or watching an extremely rapid process take place. Anything that seems overwhelming to take in triggers me to consider a blink test. By the way, when I say “blink test” I’m talking mostly about a blink oracle.
When presented with an application to test, what triggers (if any) lead you to choose model-based testing? or do you always think through state transitions?
Everyone, always, is doing model-based testing if that term means “testing according to a model.” The only possible exception to that would be testing by accident. As soon as you test purposefully, that means you already have a model in mind.
Automatically generating tests according to a specified model is the narrower definition of model-based testing that people like Google’s Harry Robinson prefer. What triggers that for me is whenever I want to meticulously cover a test space that I can conveniently describe with a tractable handful of variables (or if I see a regular and simple notation to describe those tests, regardless of the number of variables). I then write a little program in Perl to generate the test ideas. I may also try to automate those tests. For instance, I wrote a program to generate tests for the example I used in my talk. It produced 152 state transition cases, each consisting of a start state, three actions, and an expected end state. Like this:
TRANSITION SEQUENCES: 1 launching -> (finished launching) stop start -> running 2 launching -> (finished launching) stop reset -> resetted 3 launching -> (finished launching) stop stop -> stopped 4 launching -> (finished launching) reset start -> running 5 launching -> (finished launching) reset reset -> resetted ..... 148 stopped -> stop reset start -> running 149 stopped -> stop reset reset -> resetted 150 stopped -> stop stop start -> running 151 stopped -> stop stop reset -> resetted 152 stopped -> stop stop stop -> stopped
How do you suggest that you apply this technique with a complex application, something like MS Word for instance.
I’m not sure what technique you are referring to. If you are talking about using state models to describe a system, then it’s interesting that you ask that, because I asked Harry Robinson that same question (it was even about applying it to Word) after seeing an otherwise fascinating talk about state-based testing that he gave, years ago. As I recall, he wasn’t ready to answer that question. But I am. My answer is: ask Harry. Seriously, he has actually worked with model-based testing tools at Microsoft, in the years since I challenged him on this point.
I have another answer, too. I don’t apply automated model-based testing to entire applications, I apply that method opportunistically. So, if I were testing Word, I would be looking for features of Word that seemed especially like tractable state machines, or were tractable (meaning not too many variables and complications) to test via some other kind of model. Otherwise, what I am always doing is developing models in my mind (we call it learning) and using those models to test any given product, no matter how complex it is.
Have you come across any tools for automated Model based testing?
I use Perl. I’m sure there are other kinds of tools, but I haven’t used them.
What’s too much modeling?
Too much formal modeling is when you give it more time than it is worth, or when many other interesting things don’t get done because you are obsessed with the formalisms and the cool tools. Too much attention to one kind of model will starve attention for other kinds of models that also have testing value.
The technology of Webinars is still a bit immature. I used GoToWebinar, and I think the price/value arrangement is pretty good. WebEx has better features, but it’s much more expensive.
I had to wear two headsets, one to record my voice and talk to the CAMUG auditorium via Skype, the other to talk on the webinar conference line. The CAMUG people (in an auditorium) and the online people were not able to hear each other. I wasn’t able to hear the online people, but they could type questions to me. I wish the audio was all somehow integrated online. I would have wanted to record the CAMUG audio, too.
Animations happening on my screen were not smoothly displayed to the audience, but at least they were displayed.
Skype drops calls pretty often, but luckily not during this particular presentation.
Once in a long while (about every two years) Cem Kaner agrees to do a little corporate consulting and teaching. When he does, it’s through my company. Well, he tells me he’s going to be available this summer to teach his venerable Black Box Testing Class.
Considering his research agenda and university schedule, Cem is generally not available to teach anymore, outside of Florida Tech. However, he has continued to develop his testing class, much of which you can see online, now (see www.testingeducation.org).
If you don’t know Cem (http://www.kaner.com), he wrote Testing Computer Software, a book that influenced me quite a bit in the early 90’s. He was lead author for Lessons Learned in Software Testing. He has been a driving force in the development of the Context-Driven testing school– it was his idea. He also introduced the idea of small roundtable conferences (I call them peer conferences) to our industry, in ‘97, and today is pioneering the idea of free, universally available online testing training. Oh and he’s also a lawyer, so he can tell you a lot about the legal liabilities of software quality.
It’s possible that I will set up a public class for him. We haven’t decided, yet. But if you’re interested in bringing his class to your company, contact me, at james@satisfice.com.
A while ago, Mike Kelly posted two audio/video snippets of a test coaching session we did together.
See them here: http://michaeldkelly.com/media.html
I’m pleased to announce that Pradeep Soundararajan is opening a franchise of Satisfice, Inc. in India. Pradeep has been a student of mine for the last few months. I’ve been very impressed with his sheer passion for testing and, more importantly, for his passion to learn to be a better tester. You can see this clearly in his writings. He also has a particular flair for lateral thinking and brainstorming test ideas.
Having a franchise of Satisfice means that Pradeep will have me monitoring and supporting his testing and test training work. He can work much less expensively than I can, because his overhead is so much lower, my job is to assure that he is continually gaining skill so that his standard of workmanship meets my expectations.
Now, I want all of you to give Pradeep a hard time. A Satisfice tester, (as my brother Jon, at Quardev will tell you from working for me for 18 months) is expected to be quantum cut above normal testers. We achieve that not through wishful thinking, but through study and practice.
A Satisfice tester is always ready to be challenged about his work. I would expect you to ask him “How do you know this is good work, Pradeep? What methodology did you use? What alternative methodologies could you have used? What is your coverage and what are you oracles? State your test strategy on one piece of paper. Please explain testing in three minutes or less. What is rapid testing and how does that differ from context-driven testing. Hurry, Pradeep, you are supposed to be a Satisfice tester!”