“The Proposed Final Draft of the Principles of the Law of Software Contracts was approved, subject to the discussion at the meeting and to editorial prerogative. Approval of the draft clears the way for publication of the official text of this project.” (from a report to members on the  actions taken this week at the American Law Institute’s annual meeting.)

I’ll talk more about these at CAST, this summer.

This document will influence court decisions across the United States. It is the counterbalance to the Uniform Computer Information Transactions Act, which vastly increased software seller’s powers, virtually wiped out customers’ abilities to hold companies accountable for bad software—UCITA passed in 2 states, then died because it was so widely seen as so unbalanced.

The main provisions of the Principles that affect us:

(1) Companies will be required to reveal known defects at time of sale

(2) Reverse engineering will be more legally defensible. People will now have ALMOST as much right to reverse engineer software, in the United States, as they have for every other kind of product in the US. This brings us closer to international standards, making our development efforts less uncompetitive compared to most other high-tech countries.

I helped write the Principles. I wish I could give you more details about the discussion at the meeting (and will be able to by CAST). Unfortunately, I got a nasty virus last week and could not travel.

One comment. Earlier in the week, there was a lot of baloney on the web about a carefully timed letter from Microsoft and the Linux Foundation that (a) pleaded for delay because they said they needed more time to review the draft and (b) said that the disclosure requirements were very new and onerous.

Actually, the community has been aware of proposals for disclosure since 1995, when Ed Foster published widely-read articles on UCITA (then called Article 2B) in Infoworld, which were followed up by a lot of mass-media attention. There have been several follow-up reports to our community (software development, software testing) since then, including talks that I’ve given at previous CAST meetings.

In terms of awareness by LAWYERS, Microsoft has been involved in the drafting process for UCITA and the ALI Principles for longer than I have (I stated working on this in 1995; I think they started in 1989). The Open Source communities have been more variable in their activism on these laws, but several attorneys within that community have been active. More to the point, the Principles specifically exempts open source software from the disclosure rule because the distribution models (and availability of code) are so different from traditional proprietary software. The MS/Open Linux letter also complained that the ALI is treating all software transfer as if it were packaged software. This is a criticism that was applied to early drafts of UCITA (which Microsoft, Apple and IBM played heavy roles in writing) but that was pretty cleared up before UCITA was introduced to state legislatures in 2001. The ALI Principles were started after that, well after everyone in the process understood the variety of distribution models for software. Letters like this make good copy on slashdot and in blogs where authors don’t know much about the law or history of the work they’re blogging about, but as serious criticisms, they seem devoid of merit.

James, Bret and I published our definition of context-driven testing at http://www.context-driven-testing.com/.

Some people have found the definition too complex and have tried to simplify it, attempting to equate the approach with Agile development or Agile  testing, or with the exploratory style of software testing. Here’s another crack at a definition:

Context-driven testers choose their testing objectives, techniques, and deliverables (including test documentation) by looking first to the details of the specific situation, including the desires of the stakeholders who commissioned the testing. The essence of context-driven testing is project-appropriate application of skill and judgment. The Context-Driven School of testing places this approach to testing within a humanistic social and ethical framework.

Ultimately, context-driven testing is about doing the best we can with what we get. Rather than trying to apply “best practices,” we accept that very different practices (even different definitions of common testing terms) will work best under different circumstances.

Contrasting context-driven with context-aware testing.

Many testers think of their approach as context-driven because they take contextual factors into account as they do their work. Here are a few examples that might illustrate the differences between context-driven and context-aware:

• Context-driven testers reject the notion of best practices, because they present certain practices as appropriate independent of context. Of course it is widely accepted that any “best practice” might be inapplicable under some circumstances. However, when someone looks to best practices first and to project-specific factors second, that may be context-aware, but not context-driven.
• Similarly, some people create standards, like IEEE Standard 829 for test documentation, because they think that it is useful to have a standard to lay out what is generally the right thing to do. This is not unusual, nor disreputable, but it is not context-driven. Standard 829 starts with a vision of good documentation and encourages the tester to modify what is created based on the needs of the stakeholders. Context-driven testing starts with the requirements of the stakeholders and the practical constraints and opportunities of the project. To the context-driven tester, the standard provides implementation-level suggestions rather than prescriptions.

Contrasting context-driven with context-oblivious, context-specific, and context-imperial testing.

To say “context-driven” is to distinguish our approach to testing from context-oblivious, context-specific, or context-imperial approaches:

• Context-oblivious testing is done without a thought for the match between testing practices and testing problems. This is common among testers who are just learning the craft, or are merely copying what they’ve seen other testers do.
• Context-specific testing applies an approach that is optimized for a specific setting or problem, without room for adjustment in the event that the context changes. This is common in organizations with longstanding projects and teams, wherein the testers may not have worked in more than one organization. For example, one test group might develop expertise with military software, another group with games. In the specific situation, a context-specific tester and a context-driven tester might test their software in exactly the same way. However, the context-specific tester knows only how to work within her or his one development context (MilSpec) (or games), and s/he is not aware of the degree to which skilled testing will be different across contexts.
• Context-imperial testing insists on changing the project or the business in order to fit the testers’ own standardized concept of “best” or “professional” practice, instead of designing or adapting practices to fit the project. The context-imperial approach is common among consultants who know testing primarily from reading books, or whose practical experience was context-specific, or who are trying to appeal to a market that believes its approach to development is the one true way.

Contrasting context-driven with agile testing.

Agile development models advocate for a customer-responsive, waste-minimizing, humanistic approach to software development and so does context-driven testing. However, context-driven testing is not inherently part of the Agile development movement.

• For example, Agile development generally advocates for extensive use of unit tests. Context-driven testers will modify how they test if they know that unit testing was done well. Many (probably most) context-driven testers will recommend unit testing as a way to make later system testing much more efficient. However, if the development team doesn’t create reusable test suites, the context-driven tester will suggest testing approaches that don’t expect or rely on successful unit tests.
• Similarly, Agile developers often recommend an evolutionary or spiral life cycle model with minimal documentation that is developed as needed. Many (perhaps most) context-driven testers would be particularly comfortable working within this life cycle, but it is no less context-driven to create extensively-documented tests within a waterfall project that creates big documentation up front.

Ultimately, context-driven testing is about doing the best we can with what we get. There might not be such a thing as Agile Testing (in the sense used by the agile development community) in the absence of effective unit testing, but there can certainly be context-driven testing.

Contrasting context-driven with standards-driven testing.

Some testers advocate favored life-cycle models, favored organizational models, or favored artifacts. Consider for example, the V-model, the mutually suspicious separation between programming and testing groups, and the demand that all code delivered to testers come with detailed specifications.

Context-driven testing has no room for this advocacy. Testers get what they get, and skilled context-driven testers must know how to cope with what comes their way. Of course, we can and should explain tradeoffs to people, make it clear what makes us more efficient and more effective, but ultimately, we see testing as a service to stakeholders who make the broader project management decisions.

• Yes, of course, some demands are unreasonable and we should refuse them, such as demands that the tester falsify records, make false claims about the product or the testing, or work unreasonable hours. But this doesn’t mean that every stakeholder request is unreasonable, even some that we don’t like.
• And yes, of course, some demands are absurd because they call for the impossible, such as assessing conformance of a product with contractually-specified characteristics without access to the contract or its specifications. But this doesn’t mean that every stakeholder request that we don’t like is absurd, or impossible.
• And yes, of course, if our task is to assess conformance of the product with its specification, we need a specification. But that doesn’t mean we always need specifications or that it is always appropriate (or even usually appropriate) for us to insist on receiving them.

There are always constraints. Some of them are practical, others ethical. But within those constraints, we start from the project’s needs, not from our process preferences.

Context-driven techniques?

Context-driven testing is an approach, not a technique. Our task is to do the best testing we can under the circumstances–the more techniques we know, the more options we have available when considering how to cope with a new situation.

The set of techniques–or better put, the body of knowledge–that we need is not just a testing set. In this, we follow in Gerry Weinberg’s footsteps:  Start to finish, we see a software development project as a creative, complex human activity. To know how to serve the project well, we have to understand the project, its stakeholders, and their interests. Many of our core skills come from psychology, economics, ethnography, and the other socials sciences.

Closing notes

Reasonable people can advocate for standards-driven testing. Or for the idea that testing activities should be routinized to the extent that they can be delegated to less expensive and less skilled people who apply the routine directions. Or for the idea that the biggest return on investment today lies in improving those testing practices intimately tied to writing the code. These are all widely espoused views. However, even if their proponents emphasize the need to tailor these views to the specific situation, these views reflect fundamentally different starting points from context-driven testing.

Cem Kaner, J.D., Ph.D.
James Bach

Ed Foster just died.

Ed was one of the great journalists of Silicon Valley. He listened. He read. He asked probing questions. He changed his mind when the evidence proved him wrong.  He understood the computer and information industries from (at least) a dozen perspectives. And he could explain their perspectives to each other.

Ed was part of the heart of Silicon Valley in the early years of the small computer revolution. He was a whirlwind of well-informed enthusiasm. He taught us about the culture and the values of the Valley.  Consumers, hackers, publishers, marketers were all entertained and informed by him. Directly and indirectly, he shaped our thinking about the potential of this new technology and the responsibilities of these new technologists within the technology-enthusiast society and the broader American society.

I think I first met Ed in the 1980’s at one of the trade shows, but I didn’t have the privilege of talking with him in depth, then of collaborating with him, until the mid-1990’s.

Ed was the first journalist to publicize a series of projects to rewrite the commercial laws governing computers and software.

These new laws were being presented to people, mainly to the broad legal community (no one else was listening in those early days)(well, almost no one–Ed was listening) as if they were a careful balance of the rights of consumers, small business customers, small software developers, the open source community, and bigger software publishers and hardware makers. On the surface, they looked that way. Beneath that surface was a new legal regime designed to give software publishers, database publishers and large software consulting firms a panoply of new rights and defenses.

I was a newly-graduated lawyer when Ed’s comments alerted me to the new stuff on the horizon. I went to school intending to work on the law of software quality. Following up Ed’s leads shaped my career.

As Ed (with some help from me) caught a glimmer of the scope and significance of these proposals, Ed got to work. He read voraciously. He came to meetings. He interviewed and interviewed and interviewed people. He checked facts. He checked his assumptions and conclusions. He took advice from people who disagreed with him as well as from those who agreed. He wrote with care and credibility. His leadership brought dozens of other journalists into coverage of the nuts and bolts of development of highly technical commercial law–something almost never covered by the press. He helped them understand what they were seeing. He was A Force To Be Reckoned With, not because he represented people with power or money but because he did his homework and knew how to explain what he knew to ordinary readers. The most visible bills of this group were the Uniform Computer Information Transactions Act (which was ultimately rejected by 48 of 50 States) and the Uniform Electronic Transactions Act (which improved tremendously under the bright light of public scrutiny. You might know it better under its federal name, ESIGN. It governs electronic commerce in every State). Without Ed, neither result would have happened.

One of the proposals that Ed embraced with passion was the idea that software companies should be required to disclose their known defects.  There’s a natural justice in the idea that a company who knows about a bug but won’t tell its customers about it should be responsible to those customers for any losses caused by the known bug.  You can’t find every bug. But if you honestly and effectively disclose, your customers can at least work around the bugs you know about (or buy a product whose bugs are less serious). Ed could make this natural justice clear and obvious. The implementation of the idea (writing it into a set of laws) is complex–you can easily get lost in the difficult details–but Ed could stand above that and remind people why the implementation was worth the effort. I was initially a skeptic–I favored the idea but saw other matters as more critical. Ed turned my priorities around, leading me gradually to understand that there is no real competition and no hope for justice in a marketplace that allows vendors to hide fundamental information from the people who most need it.

The UCITA drafters dismissed this as naive, unreasonable, excessively burdensome, or impossible to do. But now that UCITA has failed, a major new drafting project (the American Law Institute’s Principles of the Law of Software Contracting) has picked up the idea. It will probably appear in serious legislation in 2012 or so, almost twenty years  after Ed started explaining it to people. I am sad that Ed will miss seeing this come to fruition. Among his many gifts to American society, this is an important one.

In this decade, Ed has been one of this industry’s extremely few consumer advocates. Since 2000, when consumer protection at the Federal level went dark and State-level protection continued to vanish in the never ending waves of tax cuts and litigation “reforms,” I have learned more about the pulse of consumer problems from Ed’s alerts than from any other source.

I teach courses on computer law and ethics these days, to budding software engineers. Ed’s work provided perfect starting points for many students. Some probably learned more about professionalism and ethics from Ed’s writing than from anything in their textbooks or my lectures.

Ed wrote as a voice of the conscience of an industry that needs to find its conscience.

I am writing through tears as I say that he will be missed

– Cem Kaner

This year’s theme at CAST was multidisciplinary approaches to testing. Tying my experience in psychological research, legal practice, and testing, I gave the keynote address:  The Value of Checklists and the Danger of Scripts: What Legal Training Suggests for Testers. At its core, the talk says this:

In the hands of professionals, checklists facilitate the exercise of judgment by the human professional. In contrast, scripts attempt to mechanize the task (whether a machine runs the test or a human being treated as a machine). Professionals often use checklists and, in my experience, rarely use scripts. When I went to law school, professors spoke often about reliance on script-like tools–in essence, they described them as paving stones on a fast highway to malpractice. My training and practice in psychology and law laid a foundation for my skepticism about linking scripting to “professionalism” in software testing.

Checklists (of many kinds) are fine tools to help testers prepare for exploratory testing or to document their work. Scripts are fine tools for bug reporting (”do these things to get the failure”) and necessary tools for dealing with regulators who demand them, but as test planning tools, they are an anti-professional practice.

Along with the overview, the talk provided several examples of different types of checklists, with thoughts on how to apply them to testing.

http://www.kaner.com/pdfs/ValueOfChecklists.pdf

A couple of years ago, several of us worked on definitions of exploratory testing at the Workshop on Heuristic and Exploratory Techniques.  We didn’t reach unanimous agreement at that meeting. However, later discussions based on the meeting’s notes yielded this definition:

“Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

I cannot present this as a consensus definition from WHET, both because (a) not everyone who came to WHET would agree and (b) several other people helped polish it. But I am uncomfortable presenting it as if it is primarily from James Bach, Jon Bach, Mike Bolton, and/or me. The authorship / endorsement list should be much broader.

If you would like to be identified either as one of the authors of this definition or as an endorser of it, can you please post a comment to this blog or send me an email (kaner@kaner.com)

You’ve read about the Association for Software Testing’s free software testing courses. Now find out how you can get involved in teaching these for AST, for your company, or independently. This workshop will use presentations, lectures, and hands-on exercises to address the challenges of teaching online: Becky Fiedler, Scott Barber and I will host the Live! AST Instructors’ Orientation Course Jumpstart Tutorial On July 17, 2008, in conjunction with this year’s Conference of the AST (CAST).

To register for this Tutorial, go to https://www.123signup.com/register?id=tbzrz

To register for CAST (early registration ends June 1)  go to http://www.cast2008.org/Registration

More Details

AST is developing a series of online courses, free to our members, each taught by a team of volunteer instructors. AST will grant a certificate in software testing to members who have completed 10 courses. (We hope to develop our 10th course by July 2009).

AST trains and certifies instructors: The main requirements for an AST member to be certified to teach an AST course  are (a) completing the  AST course on teaching online, (b) teaching the same course three times under supervision, (c) approval by the currently-certified instructors for that course, and agreement to teach the course a few times for AST (for free). (More details here.)

As a certified instructor, you can offer the course for AST credit: for AST (for free), for your company, or on your own. You or your company can charge fees without paying AST any royalties or other fees. (AST can only offer each free course a few times per year–if demand outstrips that supply, instructors will have a business opportunity to fill that gap.)

This Tutorial, the day after CAST, satisfies the Instructors’ Orientation Course requirement for prospective AST-certified instructors.

This workshop will use presentations, lectures, and hands-on exercises to address the challenges of teaching online. (Bring your laptop and wireless card if you can.) The presenters will merge instructional theory and assessment theory to show you how they developed the AST-BBST online instructional model.  Over lunch, Scott Barber will lead a panel discussion of AST members who are working on AST Instructor Certification.

Your registration includes a boxed lunch and light snacks in the morning and afternoon.

This workshop is partially based on research that was supported by NSF Grants EIA-0113539 ITR/SY+PE: “Improving the Education of Software Testers� and CCLI-0717613 “Adaptation  & Implementation of an Activity-Based Online or Hybrid Course in Software Testing.� Any opinions, findings and conclusions or recommendations expressed in this workshop are those of the presenter(s) and do not necessarily reflect the views of the National Science Foundation.

There are still sponsorship opportunities available for CAST08, the annual conference of the Association for Software Testing.The AST is a non-profit professional association with a mission of advancing the understanding and practice of software testing through conferences, LAWSTâ„¢ style peer workshops, publications, training courses, web sites, and other services.

This is an exciting year for CAST. We are planning for 350 participants to join us this year at 89 Chestnut in Toronto, Canada. We’ve secured Gerald M. Weinberg, Cem Kaner, Robert Sabourin, and Brian Fish as keynote speakers related to the conference theme of Beyond the Boundaries: Interdisciplinary Approaches to Software Testing. Pre-conference tutorials are being given by Jerry Weinberg, Scott Barber, Hung Nguyen, and Julian Harty. The rest of the program is equally impressive in terms of content and speaker quality. As an added bonus, Jerry has scheduled the official launch of his new book Perfect Software and Other Testing Myths to be at CAST. These are just some of the reasons that CAST attracts some of the most well known and influential testers, testing consultants, and trainers of testers from around the globe to attend as participants.

CAST’s Sponsorship and Exhibitor Programs include what you’d expect from a conference in our industry, but probably at a lower price than you might be used to.  There are packages where promotional material is delivered to participants on your behalf, meal and artifact (participant backpacks, proceedings CDs, etc.) sponsorship opportunities, as well as exhibitor booth options. Additionally, this year we have added a “Vendors & Service Providers Only” track at the conference.  This is where you get to come and discuss, demonstrate, answer questions about, and/or introduce your products and services to interested CAST participants. We do require vendors to follow the same 2/3, 1/3 rule that CAST mandates for all sessions, that is: You present for up to 2/3 of the total time slot, you leave at least 1/3 of the time for facilitated questions and answers. Last year we piloted this idea with 2 sessions. Participants *loved* that the vendors were presenting then sticking around to field hard questions and the vendors who presented reported that in addition to high quality leads and contacts, they gained significantly more valuable market information than they get as exhibitors or vendor-presenters at other conferences.

I hope you find all the information you need to decide what level of sponsorship is right for you in the link below, but if you do not find what you need, please do not hesitate to contact AST’s Executive Director and this year’s CAST Sponsorship Chairperson, Scott Barber at
executive.director@associationforsoftwaretesting.org

CAST 2008 Sponsorship and Exhibitor Program:
http://www.cast2008.org/sponsors

“LONDON, May 21 (Reuters) - A computer coding error led Moody’s Investors Service to assign incorrect triple-A ratings to a complex debt product that came to mark the peak of the credit boom, the Financial Times said on Wednesday. (see www.forbes.com/reuters/feeds/reuters/2008/05/21/2008-05-21T075644Z_01_L21551923_RTRIDST_0_MOODYS-CPDOS.html. For more, see blogs.spectrum.ieee.org/riskfactor/2008/05/moodys_rating_bug_gives_credit.html and www.ft.com/cms/s/0/0c82561a-2697-11dd-9c95-000077b07658.html?nclick_check=1 or just search for Moody’s software error.

This is the kind of stuff David Pels and I expected when we fought the Uniform Computer Information Transactions Act (UCITA) back in the 1990’s. UCITA was written as a shield for large software publishers, consulting firms, and other information publishers. It virtually wiped out liability for defects in information-industry products or services, expanded intellectual property rights well beyond what the Copyright Act and the patent laws provide, and helped companies find ways to expand their power to block reverse engineering of products to check for functional or security defects and to publicly report those defects.

UCITA was ultimately adopted only in Virginia and Maryland, rejected in all other American states, but largely imported into most states by judicial rulings (a fine example of “judicial activism”–imposing rules on a state even after its legislators rejected them. People who still whine about left-wing judicial activism are still stuck in the 1970’s).

David Pels and I wrote a book, “Bad Software” on the law of software quality circa 1998. It provides a striking contrast between software customers’ rights in the 1990s and the vastly-reduced rights we have come to expect today, along with some background on the UCITA legislation (UCITA was then called “Article 2B”–as part of a failed effort to add a new Article to the Uniform Commercial Code). John Wiley originally published Bad Software, but they have let me post a free copy at the web. You can find it at http://www.badsoftware.com/wiki/

As you’ve been able to see for a while, from his campaign picture on my blog, I support Barack Obama for U.S. president.

This blog isn’t the vehicle that I want to use for political discussion. Instructional theory, engineering practice, and the legal context (the evolution of the law of software quality) yes. Election-year politics, no.

If you’re interested in my more political views, see my new blog at http://my.barackobama.com/page/community/blog/cemkaner

The CAST program is at http://www.associationforsoftwaretesting.org/drupal/CAST2008/Program

CAST is a special conference because of its emphasis on open discussion and critical thinking.

For example, all speakers, including keynotes, are subject to unlimited questioning. If the discussion of a keynote goes beyond the end of the keynote time, we either move the next session to a new room, or move the keynote discussion to a new room. For example, the “questioning” for a keynote by James Bach included a full critical counterpresentation by one of the people in the audience, followed by a long debate. Several 1-hour track presentations have morphed into 2-hour (moderated) discussions.