Show This Video To Your Boss
Cem Kaner has produced a 16-minute video that nicely explains why excellent testing must be exploratory to some degree, and why scripted testing can’t satisfy us if we want to find important bugs fast in a changing product.
You can find the video here: http://www.satisfice.com/bbst/videos/BBSTExploring1.wmv
Demonstration and explanation videos are the Big New Thing in the context-driven community. But they take a lot of effort to produce. I’m amazed at the time Cem puts into these things.
December 28th, 2006 at 10:54 am
Any chance of streaming this on youtube or google video? 30MB is a big download…
[James’ Reply: I agree. I’m looking into it.]
December 28th, 2006 at 1:04 pm
James, this is just the first 1/6th of the exploratory testing lecture. This segment is just the introduction. It’s OK for folks to view it, but please come back for the rest, which I hope to complete and post piecemeal over the next two weeks. (Editing time takes many hours per segment.)
[James’ Reply: Yeah, I’m eager to see the next segments. We’re going to need a video hosting service for the rest.]
December 28th, 2006 at 4:20 pm
I am not sure that this reply belongs here but I’ll give it a shot.
Over the past 4 years my work environment has been influenced by the FDA.
Our testing approach(es) have been ‘guided’ by FDA publications. Testing is a combination of IQ, OQ, and PQ. (installation qualification, operational qualification, and performance qualification).
It seems to me that this terminology was initially identified for the manufacturing world, then has been applied to the world of SW testing. This is exactly what Cem talked about in his video.
In their “General Principles of Software Validation” ( http://www.fda.gov/cdrh/comp/guidance/938.pdf ) the FDA states the following:
The real effort of effective
software testing lies in the definition of what is to be tested rather than in the performance of the test.
A software testing process should be based on principles that foster effective examinations of a software
product. Applicable software testing tenets include:
· The expected test outcome is predefined;
· A good test case has a high probability of exposing an error;
· A successful test is one that finds an error;
· There is independence from coding;
· Both application (user) and software (programming) expertise are employed;
· Testers use different tools from coders;
· Examining only the usual case is insufficient;
· Test documentation permits its reuse and an independent confirmation of the pass/fail status of a
test outcome during subsequent review.
I wonder where the FDA gets these principles. IEEE might have something to do with it, as the FDA’s Glossary document points back to IEEE for a lot of its definitions.
In a recent publication(study) the FDA wonders why many of the medical devices still have a lot of bugs (resulting in recalls in some occasions). I guess they might find some answers within their own guidance.
Still fighting the battle.
Erwin.
[James’ Reply: I suspect these principles have been uncritically swept from the pages of contradictory textbooks. For instance, the notion that a successful test is one that finds a bug is a rather nutty idea that Glenford Myers asserted in his fabulously outdated 1979 book. Nobody talks about testing that way. I can’t believe Myers ever did, either. It’s a way of talking that guarantees you will be misunderstood by your clients. It’s also just wrong: I think a much stronger case can be made that a successful test is one that fulfills a useful purpose at a reasonable cost. A test that doesn’t find a bug may yet have fulfilled a purpose of looking for that bug.
I don’t know who wrote the FDA guidelines, but I would bet $100 that if I ever confronted the fellow who did, he would shrug and throw up his hands and tell me that it’s a political process and they didn’t have the time or the money to do proper research and something is better than nothing and really it’s all just common sense anyway and blah blah blah.
It’s up to us to make the testing craft. Each of us. The inertia of incompetence can be shifted and will be, when enough of us decide to raise our game.]
December 29th, 2006 at 3:00 pm
I have seen these explanation videos only on the BBST presentation created by Cem.
I am extremely pleased with the concept. It somehow represents the missing link between pure text powerpoint slides and attending a talk at a conference.
It’s a welcome alternative to presenting through Webex (or other of that kind …) presentations.
Erwin
January 3rd, 2007 at 2:40 am
Great video. Look forward to seeing another post in your blog, telling us where to find the next segments - once they become available
January 21st, 2007 at 9:19 pm
Thanks very much for sharing this - an excellent video - looking forward to additional segments about Exploratory Testing. I know what you mean about video production taking a lot of time, having experimented with Apple iMovie (easy) and Final Cut Express (powerful). We’ve tried our hand at using video to help Lydia work remotely with her granddaughters on piano technique. It is great to be able to review topics as frequently as needed, and also to come back for a refresher lecture at any time. We’ve only experimented a bit, but have seen the potential.
Video lends itself really well to some software topics that require on-screen demonstration, although standard TV resolution is a lot lower than computer screens so this has to be done carefully. We’ve worked a bit with DVD and Quicktime videos to help us to master Apple music production tools like Logic Pro.
Also - there’s James’ “show this video to your boss” suggestion - video is a tremendous tool for promotional and “evangelical” use!
Perhaps an obvious thought, but recording webinars could provide material that could be edited into production videos, with embedded slides, demos, and perhaps at someday this could make use of advanced DVD features such as multiple selectable video tracks, or audio narration (similar to Director’s Cut DVDs where technical details of a film are discussed as you view the film). One critical thing would be Digital Rights Management, to prevent unauthorized copying. Perhaps this might be marketable through something like the iTunes Music Store (here’s a prediction: the name will soon change to the “Apple Multimedia Store” or “iMedia Store” - something like that,
My work is in the cable TV industry, and I truly feel that we are only beginning to see the potential of digital video and multimedia for instructional purposes. I can imagine a hypertext document about a topic, where links would drop you into the portion(s) of a class or classes related to that topic.
It’s fun to ponder where all this might lead…
Anyway, just wanted to pass along a “Bravo!”
-john
March 26th, 2008 at 7:25 am
Hi James and Cem,
I have read your many articles and I am impressed with your work. I want to know more about exploratory testing and want to learn a lot from you.
Kindly help!!
[James’ Reply: Thanks. Sounds like I am helping.]
March 28th, 2008 at 9:39 am
Hi James,
Yes you are very helpful. James I have a question for you. In one of the lectures Dr. Kaner said that testing early a software i.e during design or requirement phase is like testing a thing which doesn’t exist. So this means that we should wait for a software to develop and then start testing. What would be your comments to it….
One more thing I am intersted in learning a lot about testing. So what should I do……….because I know u said reading books take u nowhere. ……GUIDE ME. Is there any way I can work for you.
Thanks,
Your admirer,
Vishal
[James’ Reply: Vishal, thank you for your kind words. I don’t have employees. I’m just a wandering consultant. But if you join the forum software-testing@yahoogroups.com, then you can be a part of the Great Conversation on testing.
What Cem was saying is not to delay testing until “later”, but rather that writing a lot of test cases probably isn’t going to help. Instead there are lots of other things you can do early on in a project. For instance, learning about the technology on which the product is based. Or study the potential users. Or study technical documentation. Early in projects we prepare ourselves for testing.]