I’m thinking of having badges made which say “Conscientiously Uncertified.” It’s for those of us who want to resist the dumbing down of our craft by cynical consultants promoting bogus tester certification programs.

For me, when I see that someone is certified as CSTE, ISEB, ISTQB, or CSTQE, I immediately think “there goes someone who was bullied into compliance.”

Any suggestions for what the badge would say?

Here are some options:

• Conscientiously Uncertified
• Certification Objector
• Uncertifiable
• No Bullies
• Proud to Be Uncertified

I should have a logo made, so like-minded freedom fighters can post it on their blogs. By refusing to give in to the thugs of certification, a tester shows he can follow a more difficult and more admirable path: self-education and self-certification.

Side Note: There is one certification program coming along that looks worthwhile, to me: the AST BBST certification. It will be difficult to obtain, based on demanding online coursework. It will not claim to be anything more than a certification that the tester has successfully made it through the course(s). Some of it is already available through the AST. The rest is coming. So far, the courses are free to AST members.

I will probably not be able to get this certification (I’m too disruptive in class), but at least I helped create it.

Alan Richardson writes this commentary from inside one of the stupidest of the certification programs: the ISTQB (well, he says “ISEB”, but by all accounts, it’s being taken over by ISTQB stormtroopers).

Long ago I also tried to change a certification program from the inside. I also failed. Now I do my best to cultivate the community of people who rise above it. As Alan points out, rising above can be difficult, because of all the poor fools who’ve been duped into believing that an ISTQB tester certification actually means something important.

What such certification really means is that, in England, and several other countries, certain unscrupulous or plain ignorant consultants are able to hold the testing craft for ransom, and almost no one will call them to account. Some of the perpetrators know full well what they are doing, but many of them, I think, know so little about testing that they honestly don’t realize what harm they do to the industry.

– James

My new book is done! Sort of.

I finally have a complete draft of How I Learn Stuff: Secrets of a Buccaneer-Scholar.

All I need now is a publisher.

• I had the idea for a book like this 26 years ago (original title: “School Kills”).
• I had the idea for this specific book 18 years ago.
• The oldest text in the book was written 10 years ago.
• The basic architecture and style came together 3 years ago.
• Three quarters of the book was written over the last 6 months.
• The very last section, written 3 days ago, was on the topic of….

…how procrastination is a virtue.

Jason Gorman uses a golf analogy to talk about estimation.

I like his analogy, but he didn’t take it far enough for me. He left out a key element: we may not be playing golf.

A typical sin committed by people who do studies of schedule slippages is to discuss average amounts of time to do X or Y while only considering cases where X or Y were successfully completed. What about the cancellations? Those are ignored. Being ignored, the resulting averages have questionable meaning, except to say “the people who took more than X time to do this task gave up, in our experience so far.”

Jason says that I probably won’t have to hit the golf ball 10,000 times to get it into a par 3 hole. Well, no, but if I carry it to the hole and place it in, how many hits is that? Zero? Or is it “ERROR UNDEFINED VALUE” because I cheated? This is relevant because in software development, I frequently discover that my plans won’t work as conceived, no matter how long I could work them. Or I discover a new way to do them that cuts down the time, or increases the time. Or new requirements are put on me from a technological or human source, and that messes things up. Or it turns out that the task is mooted by some deletion of requirements.

I remember the Borland C++ 4.0 project. It went through a careful planning process. We used Gantt charts. Gantt charts, I tell you! Then Microsoft shipped C++ 7.0 with a new feature, the application wizard. Our plans fell to dust. The project was restarted. A tiger team split away to create AppExpert. And those of us who were suspicious of grandiose planning for a fast-moving world got our I Told You So moment.

The nice thing about agile development is that breaking things down into smaller chunks and planning as you go makes one year slippages such as we experience in Borland C++ 4.0 far less likely. It makes the game easier to play; more stable.

So, I like Jason’s analogy. It’s a good teaching analogy because it illustrates that how you model a task dominates how you estimate it. But if I were teaching with it, I would ask the class “What assumptions am I making?” and I would get the class to make a list. Assumptions include that we know what game we are playing, we know the rules of the game, we will not be surprised by new rules, we have a clearly defined and obtainable goal, we don’t get sick or injured, etc., etc.

If I’m asked to make an estimate on which millions of dollars depend, then these are vitally important issues to raise publicly. If I’m just spit-balling an estimate in a low stress situation, then I will make a par-for-the-course guess and not worry when surprises happen.

Matt Heusser is having a peer conference. I believe it’s his first.

It’s on the topic of technical debt.

The idea of technical debt seems to be that there are tasks you are free to neglect when building a technical artifact such as software, but if you neglect those tasks, you will incur a sort of “debt” that weighs on you in the form of nagging problems that sap your productivity. Examples include neglecting unit testing, refactoring, reviews, or source control.

Paul Gerrard believes there are irrefutable testing axioms. This is not surprising, since all axioms are by definition irrefutable. To call something an axiom is to say you will cover your ears and hum whenever someone calls that principle into question. An axiom is a fundamental assumption on which the rest of your reasoning will be based.

They are not universal axioms for our field. Instead they are articles of Paul’s philosophy. As such, I’m glad to see them. I wish more testing authors would put their cards on the table that way.

I think what Paul means is that not that his axioms are irrefutable, but that they are necessary and sufficient as a basis for understanding what he considers to be good testing. In other words, they define his school of software testing. They are the result of many choices Paul has made that he could have made differently. For instance, he could have treated testing as an activity rather than speaking of tests as artifacts. He went with the artifact option, which is why one of his axioms speaks of test sequencing. I don’t think in terms of test artifacts, primarily, so I don’t speak of sequencing tests, usually. Usually, I speak of chartering test sessions and focusing test attention.

Sometimes people complain that declaring a school of testing fragments the craft. But I think the craft is already fragmented, and we should explore and understand the various philosophies that are out there. Paul’s proposed axioms seem a pretty fair representation of what I sometimes call the Chapel Hill School, since the Chapel Hill Symposium in 1972 was the organizing moment for many of those ideas, perhaps all of them. The book Program Test Methods, by Bill Hetzel, was the first book dedicated to testing. It came out of that symposium.

The Chapel Hill School is usually called “traditional testing”, but it’s important to understand that this tradition was not well established before 1972. Jerry Weinberg’s writings on testing, in his authoritative 1961 textbook on programming, presented a more flexible view. I think the Chapel Hill school has not achieved its vision, it was largely in dissatisfaction with it that the Context-Driven school was created.

One of his axioms is “5. The Coverage Axiom: You must have a mechanism to define a target for the quantity of testing, measure progress towards that goal and assess the thoroughness in a quantifiable way.” This is not an axiom for me. I rarely quantify coverage. I think quantification that is not grounded in measurement theory is no better than using numerology or star signs to run your projects. I generally use narrative and qualitative assessment, instead.

For you context-driven hounds out there, practice your art by picking one of his axioms and showing how it is possible to have good testing, in some context, while rejecting that principle. Post your analysis as a comment to this blog, if you want.

In any social activity (as opposed to a mathematical or physical system), any attempt to say “this is what it must be” boils down to a question of values or definitions. The Context-Driven community declared our values with our seven principles. But we don’t call our principles irrefutable. We simply say here is one school of thought, and we like it better than any other, for the moment.

In 1982, when I was still in high school, I read an article in Time Magazine about teenagers who worked as programmers. The article inspired me to quit school and go to work as a programmer, too. I’m writing about that as part of my book about self-education without self-discipline.

Anyway, one of the kids mentioned in that article was Eugene Volokh, who eventually went to law school and is now a professor at UCLA. Looking at his website, I stumbled into an article where he applies ideas from software testing to teaching law.

Check it out.

Volokh’s ideas are especially familiar to me, because Cem Kaner has often told me about how his ideas about scenario testing owe much to his legal training, where reasoning through the implications of complex hypothetical cases is a fundamental part of the curriculum.

Steve Swanson is very new to testing. I predict he has a great future. He has already noticed that the common idea of boundary testing is almost content-free. Michael Bolton and I do a whole session on how to turn boundary testing into a much more intellectual engaging activity. At the end of his post, he identifies one of the major weaknesses of the classic notion of boundary testing. This confirmed for me that he is a mind to watch.

Steve questions the idea of naming test techniques:

What’s the point of having names for things? To me having a name limits what you see and limits creativity. If you feel that certain things are not to be considered boundary tests, then maybe you won’t do them. Maybe you are pigeon-holing yourself into constraints that do not need to be there. Furthermore it seems that everyone has a different idea of what a ‘boundary’ test is. If that is the case then why even have a name for it?

Dear Steve,

I’ve been studying testing for 19 years, and let me tell you, a lot of things people write about it are fairy tales. This is the first reason why you are confused about what’s in a name: most people (not everyone) are also confused, and thus just copy what they see other people write, without thinking much about it.

To use an example from my own history, I used to talk about the difference between conformance testing and deviance testing. I learned this distinction at Apple Computer. For about five years I talked about them, until I one day I realized that it is an empty and unhelpful distinction. It was not a random realization, but was part of a process of systematically doubting all my terminology and assumptions about testing, in traditional Cartesian fashion. I just couldn’t find a good enough reason to retain the distinction of testing into conformance and deviance.

It looks like you are on a similar path. If you continue on it, the kind of thinking you are doing, will A) lead you to better resources to draw from about testing, and B) allow you to become one the leaders helping us out of this morass, C) ensure that you will be attacked by certain bloggers from Microsoft and elsewhere who just hate people who apply philosophical methods to such an apparently straightforward and automatable task like testing. (Speaking of philosophy of terminology, you may be interested in the Nominalism vs. Realism conversation, or how the pragmatism movement swept aside that whole debate, or how the structuralists and post-modernists study the semiotics of discourse. All these things relate to the issues you raise about terminology.)

I will tell you now what book you need to read that will help you more than any other on this planet: Introduction to General Systems Thinking, by Gerald M. Weinberg. It’s what I consider to be the fundamental textbook of software testing, yet not 1 in 100 testers knows about it.

A quick answer to your issue with names…

Terminology is useful for at least these reasons:

1. A term can be a generative tool. It can evoke an idea or a thought process that you are interested in. (This is different from using a term to classify or label something, which as you point out limits us without necessarily helping us.) An example of the generative use of terms in this way is the HAZOP process which uses “guidewords” to focus analysis. Even a generative usage is susceptible to bias, which is why I use multiple, diverse, overlapping terms.
2. A term can serve as a compact way to direct a co-worker. When I manage a test team, I establish terminology so that when I say “do boundary testing” I can expect the tester to know what I’m asking him to do without necessarily explaining every little thing. The term is thus attached to training, dialog, and shared experiences. (This needn’t be very limiting, although we must guard against settling into a rut and having only a narrow interpretation of terms.)
3. A term can serve as a label indicating the presence of a bigger story under the surface, much like a manilla folder marked “boundary testing” would be expected to hold some papers or other information about boundary testing. The danger, I think you’ve noticed, is that ignorant testers may quite happily pass folders back and forth that are duly labelled but are quite perfectly empty. You have to open the folders on a regular basis by asking “can you describe, specifically, what you did when you did ‘exploratory testing’? Can you show me?”
4. A term can hypnotize people. (I’m not recommending this; I’m warning you against it). Terminology, especially obscure terminology, is often used in testing to give the appearance of knowledge where there is no knowledge, in hopes that the client will fall asleep in the false assumption that everything is oooookkkkkkaaaayyyyyy. You appear not to be susceptible to such hypnosis. (Adam White has a similar resistance.)

I expect to see more example of skeptical inquiry on you blog, as you wrestle with testing, Steve. I hope you find, as I do, that it’s a rewarding occupation.

My vote for the World’s Most Inquisitive Tester is Shrini Kulkarni. He asks: Do you believe that “non sapient” tests exist or for that matter - any part (seems like a very negligibly small portion of entire universe of skilled human testing) of testing be “non sapient” ?

A sapient process is any process that relies for its success on an appropriately skilled human. A sapient test would therefore be a test that relies for its success on an appropriately skilled human.

Is there such a thing as a non-sapient test? My answer is yes, depending on what you consider to be part of the test.

An appropriately skilled programmer could decide to write a clever bit of test code that automatically watches for a potentially dangerous condition and throws an exception if that condition occurs, at which point an appropriately skilled human would have to make some appropriate response. In that case, there are three interesting parts to this testing:

• the test code itself (not sapient)
• the test design process (sapient)
• the process of reacting to the test result (sapient)

All three parts together would be called testing, and that would be sapient testing.
The test artifact (what a lot of people would call the test) is not sapient. However the process of understanding it and knowing how to maintain it and when to remove it is sapient.

The process of running the test is sapient, because running it necessarily means doing the right thing if it failed (or if it passed and wasn’t supposed to pass).

A lot of automation enthusiasts apparently think that the test code is the important part, and that test design and test execution management are but tiny little problems. I think the truth is that most automation enthusiasts just like making things go. They are not comfortable with testing sapiently, since probably no one has trained them to do that. They focus instead on what is, for them, the fun part of the problem, regardless of whether it’s the important part. What aids them in this delinquency is that their clients often have no clue what good testing is, anyway, so they are attracted to whatever is very visible, like pretty green bars and pie charts!

I experienced this first hand when I wrote a log file analyzer and a client simulator driver for a patent management system. During the year I managed testing on that project, this tool was the only thing that caused the CEO to come into the testing group and say “cool!” Yes, he liked the pretty graphs and Star Trekkie moving displays. While I was showing the system to my CEO, one of the testers who worked for me– a strictly sapient tester– watched quietly and then interjected a simple question: “How is this tool going to help us find any important problem we wouldn’t otherwise have found.” Being the clever and experienced test manager that I was at the time, I was able to concoct a plausible sounding response, off-the-cuff. But in truth his question rocked me, because I had become so interested in my test tool that I actually had stopped asking myself what test strategy goal I was supposed to be serving with it.

Bless that tester for recalling me to my duty.

There seems to be a lot of confusion about this.

Test automation is any use of tools to aid testing. Test automation has been around since DAY ONE of the computing industry. And never in that history has test automation been an “automatic way of doing what testers were doing before”, unless you ignore a lot of what testers actually do.

For the same reason, a space probe is not “an automatic way of doing what planetary scientists were doing before”, but rather a tool that extends the reach of what scientists can do. Test automation means extending the reach of testers.

Test automation is not at all new. What’s comparatively new is the idea of a tester. Long ago, in the late 40’s, dedicated testers were almost unknown. Programmers tested software. Throughout the sixties, papers about testing, such as the proceedings of the IFIPS conferences, almost always assumed that programmers tested the software they built. Testing was often not distinguished from debugging. As larger and more complex systems arose, the idea of dedicated software testers came into vogue. In 1972, at Chapel Hill, the first conference on testing software was held, and the proceedings of that conference show that testing was beginning to be thought of as a discipline worthy of study apart from programming.

At that conference, I think they took a wrong turn. There was much hope and enthusiasm for the future of test automation. That hope has not panned out. Not for lack of trying. More for lack of understanding.

What they didn’t understand, and what many contemporary programmers and testers also don’t get, is that good testing is an inherently a human process– not incidentally, not accidentally, but INHERENTLY. It’s highly social and psychological. The more complex software is, the more important that humans engage intellectually to identify and solve testing problems. But the Chapel Hill conference was dominated by men trained as programmers and electrical engineers, not people who professionally thought about thinking or who trained people to think.

(Who did, you ask? Jerry Weinberg. His 1965 Ph.D. thesis on problem solving is fabulous. He had written The Psychology of Computer Programming in 1970, a number of papers about testing during the sixties, including a section on testing in his 1961 book, Fundamentals of Computer Programming. He taught problem solving classes during the 60’s too, culminating in his 1974 book Introduction to General Systems Thinking. I regret that Jerry didn’t keynote at the Chapel Hill conference, but he will at the next CAST conference, in Toronto.)

The idea of a dedicated trained tester is newer than the idea of test automation, but unlike test automation, it’s an idea that hasn’t been tried on a large scale, because tester training is so terrible.

Pretending that testing is a simple technical process of making API calls doesn’t make the boogie beasts go away. They are still there, Microsoft. My wife still needs me to troubleshoot Microsoft Office, a product produced increasingly, I’m told, by programmers who work intensively on test tools because they never learned how to test. (My colleague Michael Bolton recently ran a testing class at Microsoft, though, so perhaps there is hope.)

Test automation cannot reproduce the thinking that testers do when they conceive of tests, control tests, modify tests, and observe and evaluate the product. Test automation cannot perform sapient testing. Therefore, automation of testing does NOT mean automation of the service provided by the software tester.

In summary, test automation means applying tools to testing. Test automation is ancient, non-programming testers are a newer idea, but what the industry has not yet really tried, except on a small and local scale, is systematically training the minds of testers, instead of just calling them “test engineers” or “SDETs”, giving them tools they don’t know how to use, and hoping for the best.

(BTW, I’m a programmer. I was hand-coding machine language on my Apple II before I ever heard of things called “assemblers” that automated the process. I also ran the Borland Turbo Debugger test team on the Borland C++ project, in the early 90’s. Before that I ran a test tool development team at Apple Computer. When it comes to programmer testing, GUI test automation, and non-GUI test automation, I’ve been there and done that.

It is my experience of the profound limitations of test automation that makes me a bit impatient with the way new generations of testers and programmers who seem to think no one ever thought of test tools before they came along.)