In the typical undergraduate science curriculum, most of the experimental design we teach to undergraduates is statistical. Given a clearly formulated hypothesis and a reasonably clearly understood oracle, we learn how to design experiments that control for confounding variables, so that we can decide whether our experimental effect was statistically significant. We also teach some instrumentation, but in most cases, the students learn how to use well-understood instruments as opposed to how to appraise, design, develop, calibrate and then apply them.

Our course was not so traditionally structured. In our course, each student had to propose and evaluate an investment strategy. We started with a lot of bad ideas. (Most small investors lose money. One of the attributes of our oracle is, “If it causes you to lose money, it’s probably a bad idea.”) We wanted to develop and demonstrate good ideas instead. We played with tools (some worked better than others) and wrote code to evolve our analytical capabilities, studied some qualitative research methods (hypothesis-formation is a highly qualitative task), ran pilot studies, and then eventually got to the formal-research stages that the typical lab courses start at.

Not surprisingly, the basics of designing a research program took about 1/3 of the course. With another course, I probably could have trained these students to be moderately-skilled EVALUATORS of research articles. (It is common in several fields to see this as a full-semester course in a doctoral program.)

Sadly, few CS doctoral programs (and even fewer undergrad programs) offer courses in the development or evaluation of research, or if they offer them, they don’t require them.

The widespread gap between having a little experience replicating other people’s experiments and seeing some work on a lab, on the one hand, and learning to do and evaluate research on the other hand — this gap is the home court for truthiness. In the world of truthiness, it doesn’t matter whether the evidence in support of an absurd assertion is any good, as long as we can make it look to enough people as though good enough evidence exists. Respectable-looking research from apparently-well-credentialed people is hard for someone to dispute if, as most people in our field, one lacks training in critical evaluation of research.

The new brand of snake oil is “evidence-based” X, such as “evidence-based” methods of instruction or in a recent proposal, evidence-based software testing. Maybe I’m mistaken in my hunch about what this is about, but the tone of the abstract (and what I’ve perceived in my past personal interactions with the speaker) raise some concerns.

Jon Bach addresses the tone directly. You’ll have to form your own personal assessments of the speaker. But I agree with Jon that this does not sound merely like advocacy of applying empirical research methods to help us improve the practice of testing, an idea that I rather like. Instead, the wording  suggests a power play that seems to me to have less to do with research and more to do with the next generation of ISTQB marketing.

So let me talk here about this new brand of snake oil (”Evidence-Based!”), whether it is meant this way by this speaker or not.

The “evidence-based” game is an interesting one to play when most of the people in a community have limited training in research methods or research evaluation. This game has been recently fashionable in American education. In that context, I think it has been of greatest benefit to people who make money selling mediocritization. It’s not clear to me that this movement has added one iota of value to the quality of education in the United States.

In principle, I see 5 problems (or benefits, depending on your point of view). I say, “in principle” because of course, I have no insight into the personal motives and private ideas of Dr. Reid or his colleagues. I am raising a theoretical objection. Whether it is directly applicable to Dr. Reid and ISTQB is something you will have to decide yourself, and these comments are not sufficient to lead you to a conclusion.

1. It is easy to promote forced results from worthless research when your audience has limited (or no) training in research methods, instrumentation, or evaluation of published research. And if someone criticizes the details of your methods, you can dismiss their criticisms as quibbling or theoretical. Too many people in the audience will be stuck making their decision about the merits of the objection on the personal persuasiveness of the speakers (which snake oil salesmen excel at) rather than on the underlying merits of the research.
2. When one side has a lot of money (such as, perhaps, proceeds from a certification business), and a plan to use “research” results as a sales tool to make a lot more money, they can invest in “research” that yields promotable results. The work doesn’t have to be competent (see #1). It just has to support a conclusion that fits with the sales pitch.
3. When the other side doesn’t have a lot of money, when the other side are mainly practitioners (not much time or training to do the research), and when competent research costs a great deal more than trash (see #2 and #5), the debates are likely to be one-sided. One side has “evidence” and if the other side objects, well, if they think the “evidence” is so bad,  they should raise a bunch of money and donate a bunch of time to prove it. It’s an opportunity for well-funded con artists to take control of the (apparent) high road. They can spew impressive-looking trash at a rate that cannot possibly be countered by their critics.
4. It is easy for someone to do “research” as a basis for rebranding and reselling someone else’s ideas. Thus, someone who has never had an original thought in his life can be promoted as the “leading expert” on X by publishing a few superficial studies of it.  A certain amount of this goes on already in our field, but largely as idiosyncratic misbehavior by individuals. There is a larger threat. If a training organization will make more money (influence more standards, get its products mandated by more suckers) if its products and services have the support of “the experts”, but many of “the experts” are inconveniently critical, there is great marketing value in a vehicle for papering over the old experts with new-improved experts who have done impressive-looking research that gives “evidence-based” backing to whatever the training organization is selling. Over time, of course, this kind of plagiarism kills innovation by bankrupting the innovators. For companies that see innovation as a threat, however, that’s a benefit, not a problem. (For readers who are wondering whether I am making a specific allegation about any person or organization, I am not. This is merely a hypothetical risk in an academic’s long list of hypothetical risks, for you to think about  in your spare time.)
5. In education, we face a classic qualitative-versus-quantitative tradeoff. We can easily measure how many questions someone gets right or wrong on simplistic tests. We can’t so easily measure how deep an understanding someone has of a set of related concepts or how well they can apply them. The deeper knowledge is usually what we want to achieve, but it takes much more time and much more money and much more research planning to measure it. So instead, we often substitute the simplistic metrics for the qualitative studies. Sadly, when we drive our programs by those simplistic metrics, we optimize to them and we gradually teach to the superficial and abandon the depth. Many of us in the teaching community in the United States believe that over the past few years, this has had a serious negative impact on the quality of the public educational system and that this poses a grave threat to our long-term national competitiveness.

Most computer science programs treat system-level software testing as unfit for the classroom.

I think that software testing can have great value, that it can be very important, and that a good curriculum should have an emphasis on skilled software testing. But the popular mix of ritual, tedium, and moralizing that has been passed off by some people as testing for decades has little to offer our field, and even less for university instruction. I think ISTQB has been masterful at selling that mix. It is easy to learn and easy to certify. I’m sure that a new emphasis, “New! Improved! Now with Evidence!” could market the mix even better. Just as worthless, but with even better packaging.

In its essence, exploratory testing is about learning new things about the quality of the software under test. Exploratory test automation is about using software to help with that learning.

Testing the value of investment models is an interesting illustration because it is all about the quality of the software, it is intensely automated, and none of the tests are regression tests. This is one illustration of automated exploration. I’ll point to others in my talk and paper.

THERE’S STILL TIME TO SUBMIT YOUR OWN PROPOSAL FOR A PRESENTATION. (The official deadline is March 20, but if you’re a few days late, I bet the conference committee will still read your proposal.)

Here’s the link: http://conferences.associationforsoftwaretesting.org/

What’s different about CAST is that we genuinely welcome discussion and debate. Any participant can ask questions of any speaker. Any participant can state her or his counter-argument to a speaker. We’ll keep taking questions all day. Artificial time limits are not used to cut off discussion.  After all, the point of a conference is “conferring.”

See you there…

Making a Difference Award

This award is presented to an individual who is widely recognized for work related to the interaction of computers and society. The recipient is a leader in promoting awareness of ethical and social issues in computing. The recipients of this award and the award itself encourage responsible action by computer professionals.

This is a once-in-a-lifetime award, which goes to (at most) one person per year.

ACM has printed a biographic summary that highlights the work that led to this award. We’ll probably add an interview soon.

The award presentation and talk will probably be at the Computers, Freedom & Privacy conference in San Jose in June.

I’ll probably also be talking about this work at a QAI meeting in Chicago, this May.

There are a few themes:

• Investment modeling as a new exemplar. Software testing helps us understand the quality of the product or service under test. There are generically useful approaches to test design, like quicktests and tours and other basic techniques, but I think we add our greatest value when we apply a deeper understanding of the application. Testing instructors don’t teach this well because it takes so long to build a classroom-wide understanding of an application that is complex enough to be interesting. For the last 14 months, I’ve been exploring investment modeling as a potential exemplar of deep and interesting testing within a field that many people can grasp quickly.
• Exploratory test automation. I don’t understand why people say that exploratory testing is always manual testing. Doug Hoffman and I have been teaching “high-volume” test techniques for twelve years (I wrote about some of these back in Testing Computer Software) that don’t involve regression testing or test-by-test scripting. We run these to explore new risks; we change our parameters to shift the focus of our search (to try something new or to go further in depth if we’re onto something interesting). This is clearly exploratory, but it is intensely automated. I’m now using investment modeling to illustrate this, and starting to work with Scott Barber to use performance modeling to illustrate it as well. Doug is working through a lot of historical approaches; perhaps the three of us can integrate our work, a lot of interesting work published by other folks, into something that more clearly conveys the general idea.
• Instructional design: Teaching software testing. Rebecca Fiedler, Scott Barber and I have worked through a model for online education in software testing that fosters deeper learning than many other approaches. The Association for Software Testing has been a major testbed for this approach. We’ve also been doing a lot in academic institutions, comparing notes in detail with faculty at other schools.
• The evolving law of software quality. Federal and state legislatures have failed to adopt laws governing software contracting and software quality. Because of this, American judges have had to figure out for themselves what legal rules should be applied–until Congress or the state legislatures finally get around to giving clear and constitutional guidance to the courts. This spring, the American Law Institute unanimously adopted the Principles of the Law of Software Contracts, which includes some positions that I’ve been advocating for 15 years. The set of papers below includes some discussion of the Principles. In addition, I’m kicking off a wiki-based project to update my book, Bad Software, to give customers good advice about their rights and their best negotiating tactics under the current legal regime. I’ll blog more about this later, looking for volunteers to help update the book.

Here’s the list of new stuff:

1. Cem Kaner, “Exploratory test automation: Investment modeling as an example.” [SLIDES]. ImmuneIT, Amsterdam, October 2009.
2. Cem Kaner, “Investment modeling: A software engineer’s approach.” [SLIDES]. Colloquium, Florida Institute of Technology, October 2009.
3. Cem Kaner, “Challenges in the Evolution of Software Testing Practices in Mission-Critical Environments.” [SLIDES]. Software Test & Evaluation Summit/Workshop (National Defense Industrial Association), Reston VA, September 2009.
4. Cem Kaner, “Approaches to test automation.” [SLIDES]. Research in Motion, Kitchener/Waterloo, September 2009.
5. Cem Kaner, “Software Testing as a Quality-Improvement Activity” [SLIDES]. Lockheed Martin / IEEE Computer Society Webinar Series, September 2009.
6. Rebecca L. Fiedler & Cem Kaner, “Putting the context in context-driven testing (an application of Cultural Historical Activity Theory)” [SLIDES]. Conference of the Association for Software Testing. Colorado Springs, CO., July 2009.
7. Cem Kaner, “Metrics, qualitative measurement, and stakeholder value” [SLIDES]. Tutorial, Conference of the Association for Software Testing. Colorado Springs, CO., July 2009.
8. Cem Kaner, “The value of checklists and the danger of scripts: What legal training suggests for testers.” [SLIDES]. Conference of the Association for Software Testing. Colorado Springs, CO., July 2009.
9. Cem Kaner, “New rules adopted for software contracts.” [SLIDES]. Conference of the Association for Software Testing. Colorado Springs, CO., July 2009.
10. Cem Kaner, “Activities in software testing education: a structure for mapping learning objectives to activity designs“. Software Testing Education Workshop (International Conference on Software Testing), Denver, CO, April 2009.
11. Cem Kaner, “Plagiarism-detection software Clashing intellectual property rights and aggressive vendors yield dismaying results.” [SLIDES] [VIDEO]. Colloquium, Florida Institute of Technology, October 2009.
12. Cem Kaner, “Thinking about the Software Testing Curriculum.” [SLIDES]. Workshop on Integrating Software Testing into Programming Courses, Florida International University, March 2009.
13. Cem Kaner (initial draft), “Dimensions of Excellence in Research“. Department of Computer Sciences, Florida Institute of Technology, Spring 2009.
14. Cem Kaner, “Patterns of activities, exercises and assignments.” [SLIDES]. Workshop on Teaching Software Testing, Melbourne FL, January 2009.
15. Cem Kaner & Rebecca L. Fiedler, “Developing instructor-coached activities for hybrid and online courses.” [SLIDES]. Workshop at Inventions & Impact 2: Building Excellence in Undergraduate Science, Technology, Engineering & Mathematics (STEM) Education, National Science Foundation / American Association for the Advancement of Science, Washington DC, August 2008.
16. Cem Kaner, Rebecca L. Fiedler, & Scott Barber, “Building a free courseware community around an online software testing curriculum.” [SLIDES]. Poster Session at Inventions & Impact 2: Building Excellence in Undergraduate Science, Technology, Engineering & Mathematics (STEM) Education, National Science Foundation / American Association for the Advancement of Science, Washington DC, August 2008.
17. Cem Kaner, Rebecca L. Fiedler, & Scott Barber, “Building a free courseware community around an online software testing curriculum.” [SLIDES]. MERLOT conference, Minneapolis, August 2008.
18. Cem Kaner, “Authentic assignments that foster student communication skills” [SLIDES], Teaching Communication Skills in the Software Engineering Curriculum: A Forum for Professionals and Educators (NSF Award #0722231), Miami University, Ohio, June 2008.
19. Cem Kaner, “Comments on the August 31, 2007 Draft of the Voluntary Voting System Guidelines.” Submitted to the United States Election Assistance Commission, May 2008.
20. Cem Kaner and Rebecca L. Fiedler, “A cautionary note on checking software engineering papers for plagiarism.”IEEE Transactions on Education, vol. 51, issue 2, 2008, pp. 184-188.
21. Cem Kaner, “Software testing as a social science,” [SLIDES] STEP 2000 Workshop on Software Testing, Memphis, May 2008.
22. Cem Kaner & Stephen J. Swenson, “Good enough V&V for simulations: Some possibly helpful thoughts from the law & ethics of commercial software.” [SLIDES] Simulation Interoperability Workshop, Providence, RI, April 2008.
23. Cem Kaner, “Improve the power of your tests with risk-based test design.” [SLIDES] QAI QUEST Conference, Chicago, April 2008
24. Cem Kaner, “Risk-based testing: Some basic concepts.” [SLIDES] QAI Managers Workshop, QUEST Conference, Chicago, April 2008
25. Cem Kaner, “A tutorial in exploratory testing.” [SLIDES] QAI QUEST Conference, Chicago, April 2008
26. Cem Kaner, “Adapting Academic Course Materials in Software Testing for Industrial Professional Development.” [SLIDES] Colloquium, Florida Institute of Technology, March 2008
27. Cem Kaner, “BBST at AST: Adaptation of a course in black box software testing.” [SLIDES]. Workshop on Teaching Software Testing, Melbourne FL, January 2008.
28. Cem Kaner, “BBST: Evolving a course in black box software testing.” [SLIDES] BBST Project Advisory Board Meeting, January 2008

“The Proposed Final Draft of the Principles of the Law of Software Contracts was approved, subject to the discussion at the meeting and to editorial prerogative. Approval of the draft clears the way for publication of the official text of this project.” (from a report to members on the  actions taken this week at the American Law Institute’s annual meeting.)

I’ll talk more about these at CAST, this summer.

This document will influence court decisions across the United States. It is the counterbalance to the Uniform Computer Information Transactions Act, which vastly increased software seller’s powers, virtually wiped out customers’ abilities to hold companies accountable for bad software—UCITA passed in 2 states, then died because it was so widely seen as so unbalanced.

The main provisions of the Principles that affect us:

(1) Companies will be required to reveal known defects at time of sale

(2) Reverse engineering will be more legally defensible. People will now have ALMOST as much right to reverse engineer software, in the United States, as they have for every other kind of product in the US. This brings us closer to international standards, making our development efforts less uncompetitive compared to most other high-tech countries.

I helped write the Principles. I wish I could give you more details about the discussion at the meeting (and will be able to by CAST). Unfortunately, I got a nasty virus last week and could not travel.

One comment. Earlier in the week, there was a lot of baloney on the web about a carefully timed letter from Microsoft and the Linux Foundation that (a) pleaded for delay because they said they needed more time to review the draft and (b) said that the disclosure requirements were very new and onerous.

Actually, the community has been aware of proposals for disclosure since 1995, when Ed Foster published widely-read articles on UCITA (then called Article 2B) in Infoworld, which were followed up by a lot of mass-media attention. There have been several follow-up reports to our community (software development, software testing) since then, including talks that I’ve given at previous CAST meetings.

In terms of awareness by LAWYERS, Microsoft has been involved in the drafting process for UCITA and the ALI Principles for longer than I have (I stated working on this in 1995; I think they started in 1989). The Open Source communities have been more variable in their activism on these laws, but several attorneys within that community have been active. More to the point, the Principles specifically exempts open source software from the disclosure rule because the distribution models (and availability of code) are so different from traditional proprietary software. The MS/Open Linux letter also complained that the ALI is treating all software transfer as if it were packaged software. This is a criticism that was applied to early drafts of UCITA (which Microsoft, Apple and IBM played heavy roles in writing) but that was pretty cleared up before UCITA was introduced to state legislatures in 2001. The ALI Principles were started after that, well after everyone in the process understood the variety of distribution models for software. Letters like this make good copy on slashdot and in blogs where authors don’t know much about the law or history of the work they’re blogging about, but as serious criticisms, they seem devoid of merit.

Some people have found the definition too complex and have tried to simplify it, attempting to equate the approach with Agile development or Agile  testing, or with the exploratory style of software testing. Here’s another crack at a definition:

Context-driven testers choose their testing objectives, techniques, and deliverables (including test documentation) by looking first to the details of the specific situation, including the desires of the stakeholders who commissioned the testing. The essence of context-driven testing is project-appropriate application of skill and judgment. The Context-Driven School of testing places this approach to testing within a humanistic social and ethical framework.

Ultimately, context-driven testing is about doing the best we can with what we get. Rather than trying to apply “best practices,” we accept that very different practices (even different definitions of common testing terms) will work best under different circumstances.

Contrasting context-driven with context-aware testing.

Many testers think of their approach as context-driven because they take contextual factors into account as they do their work. Here are a few examples that might illustrate the differences between context-driven and context-aware:

• Context-driven testers reject the notion of best practices, because they present certain practices as appropriate independent of context. Of course it is widely accepted that any “best practice” might be inapplicable under some circumstances. However, when someone looks to best practices first and to project-specific factors second, that may be context-aware, but not context-driven.
• Similarly, some people create standards, like IEEE Standard 829 for test documentation, because they think that it is useful to have a standard to lay out what is generally the right thing to do. This is not unusual, nor disreputable, but it is not context-driven. Standard 829 starts with a vision of good documentation and encourages the tester to modify what is created based on the needs of the stakeholders. Context-driven testing starts with the requirements of the stakeholders and the practical constraints and opportunities of the project. To the context-driven tester, the standard provides implementation-level suggestions rather than prescriptions.

Contrasting context-driven with context-oblivious, context-specific, and context-imperial testing.

To say “context-driven” is to distinguish our approach to testing from context-oblivious, context-specific, or context-imperial approaches:

• Context-oblivious testing is done without a thought for the match between testing practices and testing problems. This is common among testers who are just learning the craft, or are merely copying what they’ve seen other testers do.
• Context-specific testing applies an approach that is optimized for a specific setting or problem, without room for adjustment in the event that the context changes. This is common in organizations with longstanding projects and teams, wherein the testers may not have worked in more than one organization. For example, one test group might develop expertise with military software, another group with games. In the specific situation, a context-specific tester and a context-driven tester might test their software in exactly the same way. However, the context-specific tester knows only how to work within her or his one development context (MilSpec) (or games), and s/he is not aware of the degree to which skilled testing will be different across contexts.
• Context-imperial testing insists on changing the project or the business in order to fit the testers’ own standardized concept of “best” or “professional” practice, instead of designing or adapting practices to fit the project. The context-imperial approach is common among consultants who know testing primarily from reading books, or whose practical experience was context-specific, or who are trying to appeal to a market that believes its approach to development is the one true way.

Contrasting context-driven with agile testing.

Agile development models advocate for a customer-responsive, waste-minimizing, humanistic approach to software development and so does context-driven testing. However, context-driven testing is not inherently part of the Agile development movement.

• For example, Agile development generally advocates for extensive use of unit tests. Context-driven testers will modify how they test if they know that unit testing was done well. Many (probably most) context-driven testers will recommend unit testing as a way to make later system testing much more efficient. However, if the development team doesn’t create reusable test suites, the context-driven tester will suggest testing approaches that don’t expect or rely on successful unit tests.
• Similarly, Agile developers often recommend an evolutionary or spiral life cycle model with minimal documentation that is developed as needed. Many (perhaps most) context-driven testers would be particularly comfortable working within this life cycle, but it is no less context-driven to create extensively-documented tests within a waterfall project that creates big documentation up front.

Ultimately, context-driven testing is about doing the best we can with what we get. There might not be such a thing as Agile Testing (in the sense used by the agile development community) in the absence of effective unit testing, but there can certainly be context-driven testing.

Contrasting context-driven with standards-driven testing.

Some testers advocate favored life-cycle models, favored organizational models, or favored artifacts. Consider for example, the V-model, the mutually suspicious separation between programming and testing groups, and the demand that all code delivered to testers come with detailed specifications.

Context-driven testing has no room for this advocacy. Testers get what they get, and skilled context-driven testers must know how to cope with what comes their way. Of course, we can and should explain tradeoffs to people, make it clear what makes us more efficient and more effective, but ultimately, we see testing as a service to stakeholders who make the broader project management decisions.

• Yes, of course, some demands are unreasonable and we should refuse them, such as demands that the tester falsify records, make false claims about the product or the testing, or work unreasonable hours. But this doesn’t mean that every stakeholder request is unreasonable, even some that we don’t like.
• And yes, of course, some demands are absurd because they call for the impossible, such as assessing conformance of a product with contractually-specified characteristics without access to the contract or its specifications. But this doesn’t mean that every stakeholder request that we don’t like is absurd, or impossible.
• And yes, of course, if our task is to assess conformance of the product with its specification, we need a specification. But that doesn’t mean we always need specifications or that it is always appropriate (or even usually appropriate) for us to insist on receiving them.

There are always constraints. Some of them are practical, others ethical. But within those constraints, we start from the project’s needs, not from our process preferences.

Context-driven techniques?

Context-driven testing is an approach, not a technique. Our task is to do the best testing we can under the circumstances–the more techniques we know, the more options we have available when considering how to cope with a new situation.

The set of techniques–or better put, the body of knowledge–that we need is not just a testing set. In this, we follow in Gerry Weinberg’s footsteps:  Start to finish, we see a software development project as a creative, complex human activity. To know how to serve the project well, we have to understand the project, its stakeholders, and their interests. Many of our core skills come from psychology, economics, ethnography, and the other socials sciences.

Closing notes

Reasonable people can advocate for standards-driven testing. Or for the idea that testing activities should be routinized to the extent that they can be delegated to less expensive and less skilled people who apply the routine directions. Or for the idea that the biggest return on investment today lies in improving those testing practices intimately tied to writing the code. These are all widely espoused views. However, even if their proponents emphasize the need to tailor these views to the specific situation, these views reflect fundamentally different starting points from context-driven testing.

Cem Kaner, J.D., Ph.D.
James Bach

Ed was one of the great journalists of Silicon Valley. He listened. He read. He asked probing questions. He changed his mind when the evidence proved him wrong.  He understood the computer and information industries from (at least) a dozen perspectives. And he could explain their perspectives to each other.

Ed was part of the heart of Silicon Valley in the early years of the small computer revolution. He was a whirlwind of well-informed enthusiasm. He taught us about the culture and the values of the Valley.  Consumers, hackers, publishers, marketers were all entertained and informed by him. Directly and indirectly, he shaped our thinking about the potential of this new technology and the responsibilities of these new technologists within the technology-enthusiast society and the broader American society.

I think I first met Ed in the 1980’s at one of the trade shows, but I didn’t have the privilege of talking with him in depth, then of collaborating with him, until the mid-1990’s.

Ed was the first journalist to publicize a series of projects to rewrite the commercial laws governing computers and software.

These new laws were being presented to people, mainly to the broad legal community (no one else was listening in those early days)(well, almost no one–Ed was listening) as if they were a careful balance of the rights of consumers, small business customers, small software developers, the open source community, and bigger software publishers and hardware makers. On the surface, they looked that way. Beneath that surface was a new legal regime designed to give software publishers, database publishers and large software consulting firms a panoply of new rights and defenses.

I was a newly-graduated lawyer when Ed’s comments alerted me to the new stuff on the horizon. I went to school intending to work on the law of software quality. Following up Ed’s leads shaped my career.

As Ed (with some help from me) caught a glimmer of the scope and significance of these proposals, Ed got to work. He read voraciously. He came to meetings. He interviewed and interviewed and interviewed people. He checked facts. He checked his assumptions and conclusions. He took advice from people who disagreed with him as well as from those who agreed. He wrote with care and credibility. His leadership brought dozens of other journalists into coverage of the nuts and bolts of development of highly technical commercial law–something almost never covered by the press. He helped them understand what they were seeing. He was A Force To Be Reckoned With, not because he represented people with power or money but because he did his homework and knew how to explain what he knew to ordinary readers. The most visible bills of this group were the Uniform Computer Information Transactions Act (which was ultimately rejected by 48 of 50 States) and the Uniform Electronic Transactions Act (which improved tremendously under the bright light of public scrutiny. You might know it better under its federal name, ESIGN. It governs electronic commerce in every State). Without Ed, neither result would have happened.

One of the proposals that Ed embraced with passion was the idea that software companies should be required to disclose their known defects.  There’s a natural justice in the idea that a company who knows about a bug but won’t tell its customers about it should be responsible to those customers for any losses caused by the known bug.  You can’t find every bug. But if you honestly and effectively disclose, your customers can at least work around the bugs you know about (or buy a product whose bugs are less serious). Ed could make this natural justice clear and obvious. The implementation of the idea (writing it into a set of laws) is complex–you can easily get lost in the difficult details–but Ed could stand above that and remind people why the implementation was worth the effort. I was initially a skeptic–I favored the idea but saw other matters as more critical. Ed turned my priorities around, leading me gradually to understand that there is no real competition and no hope for justice in a marketplace that allows vendors to hide fundamental information from the people who most need it.

The UCITA drafters dismissed this as naive, unreasonable, excessively burdensome, or impossible to do. But now that UCITA has failed, a major new drafting project (the American Law Institute’s Principles of the Law of Software Contracting) has picked up the idea. It will probably appear in serious legislation in 2012 or so, almost twenty years  after Ed started explaining it to people. I am sad that Ed will miss seeing this come to fruition. Among his many gifts to American society, this is an important one.

In this decade, Ed has been one of this industry’s extremely few consumer advocates. Since 2000, when consumer protection at the Federal level went dark and State-level protection continued to vanish in the never ending waves of tax cuts and litigation “reforms,” I have learned more about the pulse of consumer problems from Ed’s alerts than from any other source.

I teach courses on computer law and ethics these days, to budding software engineers. Ed’s work provided perfect starting points for many students. Some probably learned more about professionalism and ethics from Ed’s writing than from anything in their textbooks or my lectures.

Ed wrote as a voice of the conscience of an industry that needs to find its conscience.

I am writing through tears as I say that he will be missed

– Cem Kaner

In the hands of professionals, checklists facilitate the exercise of judgment by the human professional. In contrast, scripts attempt to mechanize the task (whether a machine runs the test or a human being treated as a machine). Professionals often use checklists and, in my experience, rarely use scripts. When I went to law school, professors spoke often about reliance on script-like tools–in essence, they described them as paving stones on a fast highway to malpractice. My training and practice in psychology and law laid a foundation for my skepticism about linking scripting to “professionalism” in software testing.

Checklists (of many kinds) are fine tools to help testers prepare for exploratory testing or to document their work. Scripts are fine tools for bug reporting (”do these things to get the failure”) and necessary tools for dealing with regulators who demand them, but as test planning tools, they are an anti-professional practice.

Along with the overview, the talk provided several examples of different types of checklists, with thoughts on how to apply them to testing.

http://www.kaner.com/pdfs/ValueOfChecklists.pdf

“Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

I cannot present this as a consensus definition from WHET, both because (a) not everyone who came to WHET would agree and (b) several other people helped polish it. But I am uncomfortable presenting it as if it is primarily from James Bach, Jon Bach, Mike Bolton, and/or me. The authorship / endorsement list should be much broader.

If you would like to be identified either as one of the authors of this definition or as an endorser of it, can you please post a comment to this blog or send me an email (kaner@kaner.com)

To register for this Tutorial, go to https://www.123signup.com/register?id=tbzrz

To register for CAST (early registration ends June 1)  go to http://www.cast2008.org/Registration

More Details

AST is developing a series of online courses, free to our members, each taught by a team of volunteer instructors. AST will grant a certificate in software testing to members who have completed 10 courses. (We hope to develop our 10th course by July 2009).

AST trains and certifies instructors: The main requirements for an AST member to be certified to teach an AST course  are (a) completing the  AST course on teaching online, (b) teaching the same course three times under supervision, (c) approval by the currently-certified instructors for that course, and agreement to teach the course a few times for AST (for free). (More details here.)

As a certified instructor, you can offer the course for AST credit: for AST (for free), for your company, or on your own. You or your company can charge fees without paying AST any royalties or other fees. (AST can only offer each free course a few times per year–if demand outstrips that supply, instructors will have a business opportunity to fill that gap.)

This Tutorial, the day after CAST, satisfies the Instructors’ Orientation Course requirement for prospective AST-certified instructors.

This workshop will use presentations, lectures, and hands-on exercises to address the challenges of teaching online. (Bring your laptop and wireless card if you can.) The presenters will merge instructional theory and assessment theory to show you how they developed the AST-BBST online instructional model.  Over lunch, Scott Barber will lead a panel discussion of AST members who are working on AST Instructor Certification.

Your registration includes a boxed lunch and light snacks in the morning and afternoon.

This workshop is partially based on research that was supported by NSF Grants EIA-0113539 ITR/SY+PE: “Improving the Education of Software Testers� and CCLI-0717613 “Adaptation  & Implementation of an Activity-Based Online or Hybrid Course in Software Testing.� Any opinions, findings and conclusions or recommendations expressed in this workshop are those of the presenter(s) and do not necessarily reflect the views of the National Science Foundation.