Finding Your Own Integrity

I have a belief that I’m not going to justify– I’m simply going to say it and challenge you to look into your own experience and your own heart and see the truth of it for yourself: Your sense of identity, as a human among humans, is the most powerful force that animates and directs your choices. It is more important than sex or food or religion. It lurks behind every neurosis (including those involving sex or food or religion). As I read history and experience life, answers to the questions “Who am I? Am I a good example of what I should be?” are the prime movers of human choice throughout all of history, and the proximal cause of every war.

There are certainly exceptions to this rule: drug addiction, mental illness, or panic over a sudden, surprising, physical threat. Maybe those things have little to do with identity. Granted. I’m talking about normal daily life (and every Shakespeare play).

“I am an American. I am a human. I am a father. I am a husband. I am lovable. I am helpful. I am a tester. I am a skeptic. I am an outsider. I am dangerous. I am safe. I am honorable. I am fallible. I am truthful. I am intellectual…”  Each of these statements, for me, are reflective shards that tumble in a kaleidoscope of my identity. The models of personhood they represent comprise my moral compass. Although the pattern formed in that kaleidoscope may seem to shift with the situation, the underlying logic of my adult identity changes little with time.

That is the context for integrity.

Integrity means wholeness; the harmony and completeness of one’s identity. Practically speaking, a person with integrity is a person to lives consistently according to their avowed moral code, as opposed to someone who has no moral code, or who changes it as a matter of convenience. A person of integrity therefore creates continuity across the events of his life, and other people feel they know who they are dealing with.

The Challenge of Finding Your Integrity

Recently, in a discussion about what is reasonable for an employer to ask of a tester, a colleague felt I was trying to impose my own values onto potential employers of my students and wrote that as teachers of new testers “employment [for the testers] should be our first priority.” I disagreed sharply, writing that “our first priority is integrity.” My correspondent seemed to take offense to that.

Now, the employment-first position might be construed to imply that we should advocate robbing banks, because it is the quickest way to get money, or perhaps we should train prostitutes, because prostitution is an old and reliable industry with lots of job security for the right people. That would be absurd, but it’s also a straw man argument. I am certain no one intends to argue that any job is better than no job. Safety, legality and morality do enter into the equation.

Conversely, the integrity-first position might be cast as requiring a tester to immediately protest or resign in the face of any ethical dilemma or systemic ethical lapse, no matter how seemingly minor. This would turn most testers into insufferable, dour lawyers on their projects. We would get very little done. Who would hire such people?

These extreme positions are not very interesting, except as tools for meditating on what might be reasonable behavior. Therefore, I’d like to describe a less extreme position that I think is more defensible and workable. It goes like this:

1. Integrity is a vital and important matter. We suffer as people and society suffers when we treat it too lightly.

2. As testers and technical people, our integrity is routinely threatened by well-meaning clients and colleagues who want us to portray ourselves and the world to be a certain way, even if that isn’t strictly the truth.

3. If we never think directly about integrity, and simply trust in the innate goodness of ourselves and others, we are definitely taking this matter too lightly.

4. Integrity is not like a vase that shatters easily, and that once shattered is irretrievable. Integrity is more like an ongoing public artwork, exposed to and buffeted by the elements, sometimes damaged but always ultimately repairable (although our reputation may be another matter). Integrity is a work in progress for all of us.

5. Integrity, like education, is both personal and social. Your society judges you. It is reasonable that it does. But it is also reasonable to negotiate limits on that judgment. We spend our lives negotiating those lines, one way or another.

6. Forgiveness, although perhaps difficult and expensive to obtain, should always be available to us. (I test this by occasionally imagining my most “depraved” enemies in testing, and then imagining what they could do that would allow me to forgive them and even collaborate with them.)

7. Although integrity is our highest priority, in general, it is not the only thing that matters. We must apply wisdom and judgment so that the maintenance of integrity does not unreasonably affect our ability to survive. There is no set formula for how to do that.

8. Therefore, our practical priority must be: to learn how to think through and solve problems of survival while maintaining reasonable integrity. This itself is an ongoing project, requiring temperance and self-forgiveness.

9. New testers need to realize that they are not necessarily responsible for the quality of their work. Sometimes you will be asked to do things you don’t understand the value of, even though there may be value. In those situations, it’s okay to be compliant, as long as you are under supervision and someone competent is taking responsibility for what you do. It’s okay to watch and learn and not necessarily to make trouble. (Although, I usually did, even as a newbie.)

10. Experienced testers? Well, much is expected of you. Your clients (your non-tester colleagues and bosses) don’t know how to test, but you are supposed to. You can’t just do what you are told. That would be like letting a drunk friend drive home. Remember, someday your clients may sober up and wonder why you agreed to their stupid plan when you were supposed to be the expert.

Having laid this hopefully reasonable and workable strategy before you… I actually think the dispute between me and my correspondent, above, was not about the importance of integrity or employment at all, but rather about the specifics of the case we were debating. I should tell you what that was: whether it is reasonable for an employer to expect an entry-level tester to “write test cases.”

From a context-driven testing perspective, no practice can be declared unreasonable outside all contexts. But I do know a lot about the typical contexts of testing. I have seen profound waste, all around the industry, due to reckless and thoughtless documenting and counting of things called “test cases.” So, I don’t think that it is reasonable, generally speaking, to require young testers to write test cases. First, because “writing test cases” is what people who don’t know how to test think testers do– so, it’s usually an indicator of incompetent management. Second, because entry-level testers do not have the skills to write test cases in such a way that won’t result in a near complete waste of their client’s time and money. And third, because there are such obviously better things to do, in most cases, including learning about the product and actually testing the product.

Many people disagree with me. But I believe their attitude on this is the single most direct and vital cause of the perpetual infancy and impotency that we see in the testing industry. In other words, it’s not just a disagreement about style, it’s something that I think threatens our integrity as sincere and thoughtful testers. Casual shrugging about test case writing must be stamped out the way transfats are being outlawed in fast food. Yes, that also took years to accomplish.

Speaking of fast food…

Here’s a metaphor that might help: eating at McDonalds.

Eating at McDonalds will not kill you (well, not outright). But what if you were forced to eat at McDonalds for your work? Every day, breakfast, lunch and dinner. Nothing but McDonalds. What if it were obvious to you that eating at McDonalds was not helping you actually succeed in your work? What if instead it was clear to you that such a diet was harming your ability to get your work done? For instance, perhaps you are a restaurant reviewer, except you are almost always full of McDonalds food so you can’t ever enjoy a proper meal at a restaurant you are supposed to review? And yet your manager, who knows nothing about restaurant reviewing, insists that you maintain a McDonalds-dominated dietary regimen.

Couldn’t someone say, hey, it’s a job and you should do what you are told? Yes, they could say that. And it might be true enough at first. But over time, that diet would hurt you; over time, you would have to cope with how poorly you were doing what you believed to be your real job. You might even be criticized for missing bugs– I mean– failing to review restaurants fully, even though it’s largely due to your employer’s own unreasonable process prescriptions.

At some point you might say “enough!!” You might refuse to eat another Big Mac. From the point of view of your management and colleagues, it might look like you were risking your job just because you didn’t want to eat a hamburger. It might look crazy to them. But from your point of view, the issue isn’t the one burger, but rather the unhealthy system, slowly killing you. This breakdown comes more quickly if you happen to have a gluten allergy.

Ethics and integrity in testing is not just about following prissy little rules that many other people flout– it’s about not making yourself sick even if other people are willing to live a sickly life. This requires that you be able to judge what health and sickness means to you. Integrity is about identity health.

A Story of Quitting Even Though I Needed the Work

In 1998, I was hired by a consulting company outside of Washington D.C. I negotiated for a $30,000 sign-on bonus, and bought a house in Virginia. I was the sole breadwinner in my family, with a wife and son to support. I bought a new car, too. In short, I was “all in.”

Six months later, I quit. I had no other job to go to. I had bills due. It took me seven years to pay back my sign-on bonus, with interest (I forfeited it because I did not stay for two years). But with the help of colleagues and family over the following weeks, I made the transition to running my own business. I am most thankful for my wife’s response when I came home that night and told her I walked out on our only source of income. She shrugged and said it was surely for the best, and that something good would come of it. (I can only recommend, gentlemen, that you marry an optimist if you can.) I am also thankful to Cem Kaner, who bought me a laptop (my only computer was then owned by my employer) and said “times like these are when you discover who your true friends are.” This was fitting because it was partly because of Cem that I had previously decided never to sacrifice my professional integrity.

This illustrates one lesson about ethics: community support helps us find and maintain our integrity.

I quit because my company was insisting that I bill hours on a project that, in my opinion, was absolutely certain not to benefit from my work. The client wanted me to create fake test cases. They didn’t call them fake test cases, of course. They claimed to want real test cases; and good ones. But no product had been designed at that time! All I had access to was a general description of requirements, which in this case were literally statements of problems the product was intended to solve, with no information on how they would be solved. It was a safety-critical factory process control system, and no one could show me what it would look like or provide any examples of what specifically it might do. The only test cases I could possibly design would therefore be vague and imaginary, based on layers of soft, fluffy assumptions. The customer told me they would be happy if I delivered a document that consisted of the text of each requirement preceded by the phrase “verify that…” I told them they didn’t need a tester for that. They needed a macro.

The integrity picture was clouded, in that case, because the client believed they had to follow the “V-Model” process, which they had interpreted as demanding that I submit a test case specification document. It was a clash between the integrity of a heuristic (the V-Model) vs. the integrity of solving the problem for which the heuristic was designed. My client might have said that I was the one violating the integrity of the process. Whereas I would have said that my client was not competent to make that judgment.

I’m not saying I won’t do bad work… I’m just saying I won’t do bad work for money. If I do bad work, I want it to be for fun or for learning, but not to anyone’s expense or detriment. Hence a line I use once in a while “I could do that for you, except that you pay me too much.” This is one reason I like being independent. I control what I bill for, and if I think a portion of my work is not acceptable, I don’t charge for it– like a chef who refuses to serve an overcooked steak.

It wasn’t as sudden as it looked…

I didn’t just lose my temper at the first sign of trouble. Things had been coming to a boil for a while. On my very first day I reviewed the RFP for that project and concluded it was doomed, but management bid on it anyway, telling me I needed to “be practical” and that surely “we could be helpful to them if they hired us.” I needed the job, so I relented against my better judgment.

During my first staff meeting, my first week on the job, I challenged the consulting staff about what they did to study testing on their own time. My challenge was met with an awkward silence, after which one of the consultants, sounding soul-wounded, told me he was offended that I would suggest that they weren’t already good enough as testers, “These are the best people I’ve ever worked with” said the twenty-something tester with little experience and no public reputation. “But how do you know they are good?” I asked, knowing that our company had just issued a press release about having hired me (a “distinguished industry pioneer” to quote it exactly). There were other murmurs of annoyance around the table, and the manager stepped in to change the subject. I could have pushed the issue, but I didn’t. I needed the job, so I relented against my better judgment.

I was later told that despite my company’s public position, the other consultants felt that I was a mere armchair expert, whereas they were practical men. I don’t know what evidence they had for that. They never showed me what they could do that I supposedly could not. Management tolerated this attitude. That means they were lying directly to their customers about me– claiming I was an expert when clearly they did not believe I was one. I could have insisted they behave in accordance with their public statements about me. But… I needed the job, so I relented against my better judgment.

I knew the day had come when I must quit because I found myself fantasizing about throwing chairs through windows. That triggered a sort of circuit-breaker of judgment: change your life now, now, now.

So what happened after that?

I suffered for this decision. First came the panic attack. I felt dizzy and it was hard to breathe for a few hours. This was followed by a few years of patching together a project here and a project there, never more than 8 weeks from completely running out of money and credit. We were twice within a week of filing for bankruptcy in the early days. During that time I walked away from a few more projects. I resigned from a dysfunctional government project, hopefully saving valuable taxpayer dollars by not writing a completely unnecessary Software Configuration Management plan that no one on the team wanted. I got myself fired from a project at Texas Instruments after about 90 minutes, because I told them a few things they didn’t want to hear (but that I felt were both true and important).

It’s not all suffering, of course. I once was fired from a project (along with the rest of the test team) and then was the only one hired back– partly because the client realized that my high standards meant that I billed far fewer hours than other consultants. In other words, saying no and being a troublemaker earned me another 500 hours of work, while the yes-sayers lost their situations. I also got some great gigs, including my very first one as an independent, specifically because I am a rabble-rousing kind of thinker.

These days, I cultivate as many clients as I can, so that I don’t rely too much on any one of them. And I have established a reputation for being honest and blunt that generally prevents the wrong sort of people from trying to hire me. It’s not easy, but it can be done: I have made integrity my top priority.

What about before I was well known?

Well, I’ve always had this attitude. It’s not some luxury to me. It’s fundamental. That’s why I had to leave high school. I’ve never been able to “play the game” at the expense of feeling like a good honest man. Like I said, I suffered for it. I wanted to go try myself at MIT, where my much more pliable best friend from high school eventually graduated. I am born to be an academic, but since I can’t stand the compliance-to-ceremony culture of the academic world, I must be an independent scholar, without access to expensive journals and fantastic libraries.

Before anybody heard of me, I treated getting work like dating: be a slightly exaggerated version of myself so that I will be rejected quickly if the relationship is not a fit (a stress testing strategy, you might say). My big break came at Apple, where I worked for a man of vision and good humor who seemed to relish being the mentor I needed. The environment was open and supportive. There was an element of luck in that my first ten years in testing I worked for people who didn’t ask me to tell lies or do bad work on purpose.

So I know it’s possible to find such people. They are out there. You don’t have to work for bozos, and if you currently do, there is yet hope.

A person who does not live true to himself feels sick and weak inside. My identity as “excellent software tester” demands that I take my craft seriously. I hope you will take this craft seriously, too.

P.S. What if my sense of identity doesn’t require me to be good at my job?

Then, technically, none of this applies to you. Your ethical code can include doing bad work. But… why are you reading my blog? How did you get in? Guards! Seize him!

 

Justifying Real Acceptance Testing

This post is not about the sort of testing people talk about when nearing a release and deciding whether it’s done. I have another word for that. I call it “testing,” or sometimes final testing or release testing. Many projects perform that testing in such a perfunctory way that it is better described as checking, according to the distinction between testing and checking I have previously written of on this blog. As Michael Bolton points out, that checking may better be described as rejection checking since a “fail” supposedly establishes a basis for saying the product is not done, whereas no amount of “passes” can show that it is done.

Acceptance testing can be defined in various ways. This post is about what I consider real acceptance testing, which I define as testing by a potential acceptor (a customer), performed for the purpose of informing a decision to accept (to purchase or rely upon) a product.

Do we need acceptance testing?

Whenever a business decides to purchase and rely upon a component or service, there is a danger that the product will fail and the business will suffer. One approach to dealing with that problem is to adopt the herd solution: follow the thickest part of the swarm; choose a popular product that is advertised or reputed to do what you want it to do and you will probably be okay. I have done that with smartphones, ruggedized laptops, file-sharing services, etc. with good results, though sometimes I am disappointed.

My business is small. I am nimble compared to almost every other company in the world. My acceptance testing usually takes the form of getting a trial subscription to service, or downloading the “basic” version of a product. Then I do some work with it and see how I feel. In this way I learned to love Dropbox, despite its troubling security situation (I can’t lock up my Dropbox files), or the fact that there is a significant chance it will corrupt very large files. (I no longer trust it with anything over half of a gig).

But what if I were advising a large company about whether to adopt a service or product that it will rely upon across dozens or hundreds or thousands of employees? What if the product has been customized or custom built specifically for them? That’s when acceptance testing becomes important.

Doesn’t the Service Level Agreement guarantee that the product will work?

There are a couple of problems with relying on vendor promises. First, the vendor probably isn’t promising total satisfaction. The service “levels” in the contract are probably narrowly and specifically drawn. That means if you don’t think of everything that matters and put that in the contract, it’s not covered. Testing is a process that helps reveal the dimensions of the service that matter.

Second, there’s an issue with timing. By the time you discover a problem with the vendor’s product, you may be already relying on it. You may already have deployed it widely. It may be too late to back out or switch to a different solution. Perhaps your company negotiated remedies in that case, but there are practical limitations to any remedy. If your vendor is very small, they may not be able to afford to fix their product quickly. If you vendor is very large, they may be able to afford to drag their feet on the fixes.

Acceptance testing protects you and makes the vendor take quality more seriously.

Acceptance testing should never be handled by the vendor. I was once hired by a vendor to do penetration testing on their product in order to appease a customer. But the vendor had no incentive to help me succeed in my assignment, nor to faithfully report the vulnerabilities I discovered. It would have been far better if the customer had hired me.

Only the accepting party has the incentive to test well. Acceptance testing should not be pre-agreed or pre-planned in any detail– otherwise the vendor will be sure that the product passes those specific tests. It should be unpredictable, so that the vendor has an incentive to make the product truly meet its requirements in a general sense. It should be adaptive (exploratory) so that any weakness you find can be examined and exploited.

The vendor wants your money. If your company is large enough, and the vendor is hungry, they will move mountains to make the product work well if they know you are paying attention. Acceptance testing, done creatively by skilled testers on a mission, keeps the vendor on its toes.

By explicitly testing in advance of your decision to accept the product, you have a fighting chance to avoid the disaster of discovering too late that the product is a lemon.

My management doesn’t think acceptance testing matters. What do I do?

1. Make the argument, above.
2. Communicate with management, formally, about this. (In writing, so that there is a record.)
It is up to management to make decisions about business risk. They may feel the risk is not worth worrying about. In that case, you must wait and watch. People are usually more persuaded by vivid experiences, rather than abstract principles, so:
1. Collect specific examples of the problems you are talking about. What bugs have you experienced in vendor products?
2. Collect news reports about bugs in products of your vendors (or other vendors) that have been disruptive.
3. In the event you get to do even a little acceptance testing, make a record of the problems you find and be ready to remind management of that history.

 

To The New Tester

About once a week, I get an email like one of these:

Hi James,

I’m from Hyderabad, India. I’m working as a Testing Engineer and doing Manual Testing from the Last 1 Year. I want to know what are things I need to follow to become a good tester. I didn’t have any programming background. I want to learn automation testing for career growth, I want to improve my test case preparing skills and documentation skills. Please suggest me in this regard.

Hi, James,

I am looking for software testing jobs. Actually I’m fresher, I don’t know much about testing but I interested too. Can you please suggest me which type of testing is good?(like manual or automation or etc..) I have got some ideas about manual testing and DB-PL/SQL.

Respected Sir,

I am a Big Fan Of You.

BTech Graduate In Electrical & Electronics. Done Software testing course in QTP and Selenium. I got job as Mobile Application Tester. I am the only tester in this company. iOS and Android Apps. Now i have 3 months experience. I am mainly doing Manual testing. I dont know , whether i am going in right path.

Can you help me please? (Advice)

This blog post shall be my standard answer to these emails…

Dear New Tester/Testing Hopeful,

Thank you for choosing testing. We need smart people in our craft, and I hope you are one of them. I know you must have something going for you because, unlike most young testers, you reached out to me. That’s not easy.

Here are some general ideas:

  • Read my blog (www.satisfice.com/blog).
  • Read the materials on my website (www.satisfice.com).
  • Consider reading either of of my books: Lessons Learned in Software Testing, or Secrets of a Buccaneer-Scholar.
  • Read the materials and blog at www.developsense.com
  • Get on Twitter and watch the conversations among the Context-Driven community. Participate in those discussions.
  • Join software-testing@yahoogroups.com. It’s a quiet, moderated email forum for Context-Driven testers.
  • I offer free coaching, too, but you have to read this blog post first.
  • Practice testing things that aren’t secret, and then post your test results online so that others can see them.
  • Consider taking the RTI Online class.
  • Consider taking the BBST class.
  • Participate in sessions conducted by www.weekendtesting.com.
  • Consider attending the Let’s Test conference or the CAST conference.
  • If you want to get good at using tools while testing, try learning Python. Personally, I use Perl, but I hear Python is good.

Here are some books to read:

  • Introduction to General Systems Thinking, by Gerald M. Weinberg
  • Quality Software Management, Vol. 1: Systems Thinking, by Gerald M. Weinberg
  • Tacit and Explicit Knowledge, by Harry Collins
  • The Black Swan, by Nassim Taleb
  • Testing Computer Software, by Cem Kaner, remains a good classic testing book.

What about certification?

  • Don’t get certified. There are no respectable commercial testing certifications.
  • If you are forced to get certified for some reason, do not take it seriously. It’s not an achievement, it’s just a conveyor belt that extracts your money and gives you nothing you couldn’t get for the price of a Google search.
  • True certification remains this: the respect of respectable people.

What kinds of testers are there?

  • Automated? Manual? There is no such thing as manual or automated testing. It’s all just testing. Testing is often supported by tools that attempt to simulate user interaction with the system. This is what people call “test automation” even though it is only automating a crude approximation of one aspect of testing. If you have the ambition to be a one-man test team, it is extremely valuable to learn how to make your own tools.
  • Exploratory? Scripted? There is no such thing as an exploratory or scripted tester. All good testing is exploratory to some degree and scripted to some degree.
  • Tester. This is a testing generalist who can contribute to any test team. Sometimes called a QA analyst, QA engineer, or test engineer. I prefer the simplicity of “tester.”
  • Omega Tester. The omega tester (which I sometimes call a test jumper, after the analogy of a paratrooper) is one who can do anything. An omega tester is equipped to be the only tester in a project team, if necessary. Omega testers can lead testing, or work with a team of other testers. I am an omega tester. I aspire to be a good one.
  • Performance Tester. The performance tester understands the mathematics and dynamics of the performance of large-scale systems. They use tools that create high loads and measure the performance envelope of systems as they scale up. Performance testers often don’t think of themselves as testers.
  • Usability Tester. The usability tester is a bit mythical. I have met only two dedicated usability testers in my entire career, but I have seen more of them from a distance. A usability tester specializes in studying how users feel about using and learning a product.
  • Security Tester. Security testers also often don’t think of themselves as testers. Security is an exciting, specialized form of testing that requires the mastery of a great many facts about a great many technologies.
  • Testing Toolsmith. A testing toolsmith is a programmer dedicated to writing and maintaining tools that help testers. This is what a lot of people would call an “automated tester” but you better not use that term around me.
  • “SDET” Software Development Engineer in Test. This means a full on programmer who does testing using his programming skills.

I want to help everyone, but at the same time, I have my own work to do. So, I limit my attention to a special kind of student. I work with people who study and practice and want to test better and understand testing more deeply than others. If that describes you, then instead of writing to me with a question, write to me with examples of your work.

Good luck!

— James

P.S. See also what Allen Johnson has to say about this.

 

Tyranny of the Innocents, Exhibit A: Gary Cohen

Gary Cohen, Deputy Administrator and Director, Center for Consumer Information and Insurance Oversight, Centers for Medicare and Medicaid
Services, is not a computer guy. He’s a lawyer. He knows the insurance industry. And yet he was put in charge of a very large and important project: Healthcare.gov.

Here’s what BusinessWeek said about him on August 22nd:

“Gary Cohen seems awfully calm for a man whose job is to make sure Obamacare doesn’t flop. As head of CCIIO (awkward pronunciation: Suh-SIE-O), he oversees the complex, politically fraught system of state health insurance exchanges that will begin signing up uninsured Americans starting on Oct. 1. It hasn’t exactly been a smooth rollout. Many Americans still have no idea the exchanges exist, and the administration has struggled to explain who’s eligible for coverage under the Affordable Care Act and how they enroll. Cohen is convinced the confusion will clear up once things are up and running. “We’re going to get to the point where the discussions we’re having today will fade into the background,” he says.

He should have known that the system wasn’t going to work, at that point. But he’s not a technology guy, so perhaps he thought some big-brained hacker from the movies was going to pull it together at the last minute?

Here’s what he was asked and what he answered at a House Committee on Oversight hearing on May 21st:

Ms. DUCKWORTH. …Could you speak a little bit on the Administration’s readiness to
reach out to this huge number of people so that they can enroll in
time? Basically, you say that you are going to be ready to go on
October 1st, and you need to be. If not, what do you need in order
to get ready and have a successful rollout of these provisions?

Mr. COHEN. So we have a plan in place that basically is timed
so that people are getting the information close to the time in
which there is something that they can do with it. So right now we
are in what we call the education phase, which began in January
and proceeds through June, where we are just putting out information.
We are in the process of re-purposing the HealthCare.gov site
to be really a consumer information site. Our call center will be
going live in June, where people will be able to call and get information
that way. And then starting in the summer we will begin
what we call the anticipation, or get ready phase. And I am not an
expert in these things, but what I understand is that if you start
too early and then people say, well, what do I do, and then there
is nothing that they can do because it is too soon, then you may
end up having people who get a little bit kind of frustrated or disappointed.
So we really are gearing towards making sure the people get the
information they need in time for October, when they actually can
take action and begin to get enrollment coverage.

Hmm. He was asked directly if he needed anything to make sure he was ready to go on October 1st. His answer was basically: no thank you.

Did he really think everything was on track? Why didn’t his people prepare him to set expectations better?

Mr. GOSAR. Mr. Cohen, how closely is HHS working with IRS on Obamacare
implementation?
Mr. COHEN. We are working closely with IRS on those aspects of
implementation where we have to work together, so, for example,
as you know, in determining whether a person is eligible for Medicaid
or CHIP on the one hand, or tax credits in the marketplaces
on the other, income is a test, and we are working with IRS on
verifying people’s income when they apply.
Mr. GOSAR. So the IRS is going to be gathering and sending this
enormous amount of taxpayer information to all the 50 exchanges.
All 50 exchanges are to be ready by October 1st, right?
Mr. COHEN. Yes.
Mr. GOSAR. So will there be any problems with this massive
amount of data sharing?
Mr. COHEN. No. And data sharing may not be exactly the right
way to look at it. Basically what will happen is people will put information
about their income in an application; that information
will be verified by data that comes from the IRS, but there is no
exchange of information from the IRS to the exchange; the information
goes out, it is verified, and it comes back.
Mr. GOSAR. But it is still from the exchange going to the IRS,
and that is where I am going.
Mr. COHEN. It is going to the data hub. Information is coming
from the IRS to the data hub and from the exchange to the data
hub, and there is a comparison and then there is an answer back.
But the tax information isn’t actually going to the exchange.

What a refreshingly blunt answer to the question of whether there will be any trouble with data exchange: No. Unfortunately, we now know there are massive problems with that. Why didn’t he give a more nuanced answer? Why didn’t he hedge? This is why I think he’s an innocent– a child put in charge of the chocolate factory. He didn’t need to be, but that’s how he played it. I guess he was distracted by other duties and trusted the technologists? Or maybe he dismissed the concerns of the technologists as mere excuses? I wonder.

 

 

 

 

Healthcare.gov and the Tyranny of the Innocents

The failure of Healthcare.gov is probably not because of sinister people. It’s probably because of the innocents who run the project. These well-intentioned people are truly as naive as little children. And they must be stopped.

They are, of course, normal intelligent adults. I’m sure they got good grades in school– if you believe in that sort of thing– and they can feed and clothe themselves. They certainly look normal, even stately and wise. It’s just that they are profoundly ignorant about technology projects while being completely oblivious to and complacent about that ignorance. That is the biggest proximal cause of this debacle. It’s called the Dunning-Kruger syndrome (which you can either look up or confidently assure yourself that you don’t need to know about): incompetence of a kind that makes you unable to assess your own lack of competence.

Who am I talking about? I’m talking to some extent about everyone above the first level of management on that project, but mostly I’m talking about anyone who was in the management chain for that project but who has never coded or tested before in their working lives. The non-technical people who created the conditions that the technical people had to work under.

I also blame the technical people in a different way. I’ll get to that, below.

How do I come to this conclusion? Well, take a look at the major possibilities:

Maybe it didn’t fail. Maybe this is normal for projects to have a few glitches? Oh my, no. Project failures are not often clear cut. But among failures, this one is cut as clearly as the Hope Diamond. This is not a near miss. This is the equivalent of sending Hans away to sell the family cow and he comes back with magic beans. It’s the equivalent of going to buy a car and coming back with a shopping cart that has a cardboard sign on which someone has written “CAR” in magic marker. It’s a swing and a miss when the batter was not even holding a bat. It’s so bad, I hope criminal charges are being considered. Make no mistake, the people who ran this project scammed the US government.

Did it fail because it’s too hard a project to do? It’s a difficult project, for sure. It may have been too hard to do under the circumstances prescribed. If so, then we should have heard that message a year ago. Loudly and publicly. We didn’t hear that? Why? Could it have been that the technical people kept their thoughts and feelings carefully shrouded? That’s not what’s being reported. It’s come out that technical people were complaining to management. Management must have quashed those complaints.

Did politics prevent the project from succeeding? No doubt that created a terrible environment in which to produce the system. So what? If it’s too hard, just laugh and say “hey this is ridiculous, we can’t commit to creating this system” UNLESS, of course, you are hoping to hide the problem forever, like a child who has wet the bed and dumps the sheets out the back window. I suppose it’s possible that Republican operatives secretly conspired to make the project fail. If so, I hope that comes out. Doesn’t matter, though. Management could still have seen it coming, unless the whole development team was in on the fix.

Were the technical people incompetent? Probably. It’s likely that many of the programmers were little better than novices, from what I can tell by looking at the bug reports coming through. It was a Children’s Crusade, I guess. But again, so what? The purpose of management, at each of the contracting agencies and above them, is to assess and assure the general competence and performance of the people working on the job. That comes first. I’m sure there were good people mixed in there, somewhere. I have harsh feelings for them, however. I would say to them: Why didn’t you go public? Why didn’t you resign? You like money that much? Your integrity matters that little to you?

Management created the conditions whereby this project was “delivered” in a non-working state. Not like the Dreamliner. The 787 had some serious glitches, and Boeing needs to shape that up. What I’m talking about is boarding an aircraft for a long trip only to be told by the captain “Well, folks it looks like we will be stuck here at the gate for a little while. Maintenance needs to install our wings and engines. I don’t know much about aircraft building, but I promise we will be flying by November 30th. Have some pretzels while you wait.”

Management must bear the prime responsibility for this. I’m not sure that Obama himself is to blame. Everyone under him though? Absolutely.

What About Testing?

Little testing happened on the site. The testing that happened seems to have confirmed what everyone knew. Now this article has come out, about what’s happening behind the scenes. I sure hope they have excellent Rapid Testers working on that, because there is no time for TDD or much of any unit testing and certainly no time to write bloated nonsensical “test case specs” that usually infect government efforts like so much botfly larvae.

Notice the bit at the end?

“It’s a lot of work but people are committed to it. I haven’t heard anyone say it’s not a doable job,” the source said of the November 30th deadline to fix the online portal to purchase insurance on the federal exchange.

Exactly. That’s exactly the problem, Mr. Source. This is what I mean by the tyranny of the innocents. If no one is telling you that the November 30th deadline is not doable, and you think that’s a good sign, then you are an innocent. If you are managing to that expectation then you are a tyrant. It’s probably not doable. I already know that this can’t possibly leave enough time for reasonable testing of the system. Even if it is doable, only a completely dysfunctional project has no one on it speaking openly about whether it is doable.

What Can Be Done?

Politics will ruin everything. I have no institutional solution for this kind of problem. “Best practices” won’t help. Oversight committees won’t help. I can only say that each of us can and should foster a culture of personal ethical behavior. I was on a government project, briefly, years ago. I concluded it was an outlandish waste of taxpayer money and I resigned. I wanted the money. But I resigned anyway. It wasn’t easy. I had car payments and house payments to make. Integrity can be hard. Integrity can be lonely. I don’t always live up to my highest ideals for my own behavior, and when that happens I feel shame. The shame I feel spurs me to be better. That’s all I’m hoping for, really. I hope the people who knew better on this project feel shame. I hope they listen to that shame and go on to be better people.

I do have advice for the innocents. I’ll speak directly to you, Kathy Sebelius, since you are the most public example of who I am talking about…

Hi Kathy,

You’re not a technology person. You shouldn’t have to be. But you need people working for you who are, because technology is opaque. It may surprise you to know that unlike building bridges and monuments, the status of software can be effectively hidden from anyone more than one level above (or sideways from) the programmer or tester who is actually working on that particular piece of it. It’s like managing a gold mine without being able to go down into the mine yourself.

This means you are in a weak position, as an executive. You can pound the table and threaten to fire people, sure. It won’t help. The way in which an executive can use direct power will only make a late software project even later. Every use of direct power weakens your influence. Use indirect power, instead. Imagine that you are taming wild birds. I used to do that as a kid in Vermont. It requires quietness and patience. The first part is to stand for an hour holding birdseed in your hand. Stand quietly and eventually they are landing in your hand.

To have managed this project well, you needed to have created an environment where people could speak without fear. You needed to work with your direct reports to make sure they weren’t filtering out too much of the bad news. You needed to visit the project on a regular basis, and talk to the lowest level people. Then you needed to forgive their managers for not telling you all the bad news. It’s a maddeningly slow process. If you notice, the Pope is currently doing something very similar. Hey, I’m an atheist and yet I find myself listening to that guy. He’s a master of indirect leadership.

You did have the direct power to set expectations. I’m sure you realize you could have done a much better job of that, but perhaps you felt fear, yourself. As your employer (a taxpaying citizen), I bear a little of that responsibility. The country is getting the Healthcare.gov site that it deserves, in a sense.

If you are going to continue in public service, please do yourself and all of us a favor and take a class on software project management. Attend a few lectures. Get smart about what kinds of dodges and syndromes contractors use.

Don’t be an innocent, marching to the slaughter, while millions of dollars line the pockets of the people who run CGI and all those other parasite companies.

— Sincerely, James

My Political Agenda

I have $200,000 of unpaid medical bills due to the crazy jacked up prices and terrible insurance situation for individual citizens in the United States. I am definitely a supporter of the concept of health care reform, even the flawed Obamacare system, if that’s the best we can do for now.

I was pleased to see the failure of the Healthcare.gov website, at first. A little failure helps me make my arguments about how hard it is to do technology well; how getting it right means striving to better ourselves, and no formula or textual incantation will do that for us.

This is too much failure! I want it to stop now. Still, I’m an adult, a software project expert and not in any way an innocent. I know it’s not going to be resolved soon. No Virginia, there won’t be a Healthcare.gov website this Christmas.

Addendum:

From cnn.com:

Summers wrote a memo to the President in 2010 suggesting that HealthCare.gov was not something the government could handle and he needed to bring in experts.

While Summers would not provide details about internal discussions, he said Tuesday, “You need experts. You need to trust but you need to verify. You can’t go rushing the schedule when you get behind or you end up making more errors.”

Damn straight. If this is true then I’m sure glad someone around Obama had basic wisdom. I guess nobody listened to him.

A Public Service Announcement About Exploratory Testing

[Updated: I revamped and added some more examples to the list.]

I got this message from Oliver Vilson, today:

Oliver V.: hi James. Just had a chat with Helena_JM. She reminded me something… don’t know if you’ve written blog about it or pushed it into  RST.. One Test lead from another company mentioned me he has problems with his testers. Some of them are saying that they don’t have to do test plans, since your teaching seems to align that…
James Bach: Any more details?
Oliver V.: rough translation from test team lead : “ET seems to have reputation as “excuse for shitty testing”. People can’t explain what they did and why. If you ask them for test plan or explanation, all you get is “but Bach said…”.

I have, from time to time, heard rumors that some people cite my writings and teachings as an excuse to do bad testing. I think it would help to do a public service announcement…

Attention Testers and Managers of Testers

If a tester claims he is justified in doing bad work because of something I’ve published or said, please email me at james@satisfice.com, or Skype me, and I will help you stop that silliness.

I teach skilled software testing for people who intend to do an excellent job. That process is necessarily exploratory in nature. It also necessarily will have some scripted elements– partly due to the nature of thinking and partly due to the requirements of excellent intellectual work.

I do not teach evasiveness or obscurantism. I do not ever tell a tester that he can get away with refusing to explain his test process. Explaining testing is an important part of being a professional.

Why People Get Confused

I reinvented software testing for myself, from first principles. So, I teach from a very different set of premises. This is necessary, because common ideas about testing are so idiotic. But it does result in confusion when my ideas are taken out of context and “mixed in” to the idiocy. Consider: “I won’t create a detailed test plan document” is a perfectly ordinary and potentially reasonable thing to say in RST. It is a statement about things made explicit in a document, not a statement about lack of planning. Yet Factory School methodology confuses documents for content. If you say that to one of them, it may be mistaken for a refusal to apply appropriate rigor to your work.

Here are some examples of how someone might misapply my teachings:

  1. Rapid Software Testing methodology (RST) is not the same thing as exploratory testing. ET is very simple. Anyone can do ET, just as anyone can look at a painting. But there’s a huge difference between a skilled appraisal of a painting by an expert and a bored glance by a schoolkid. RST is a methodology for doing testing (including scripted and exploratory testing) well. Therefore, anyone doing ET badly is not doing my methodology.
  2. In RST, a plan is not a document, it’s a set of ideas. Therefore, I say you don’t need to have a test plan template, or any sort of written test plan document in order to have a good test plan. I often document my test ideas, though, in different ways, when that helps. Therefore, the lack of a test plan (a guiding set of ideas) probably represents an immature and possibly inadequate test process, but the lack of a test plan document is not necessarily a problem.
  3. In RST, a test is not a document, it’s a performance. Therefore the lack of documented tests is not necessarily a problem, but poor testing (which can be determined by direct observation by a skilled tester or test manager, just as poor carpentry or poor doctoring can be detected) is a problem.
  4. In RST, we have no templates for reporting. But reporting is crucial. Reporting skills are crucial. Accountability is crucial. Credibility is crucial. We teach the art of telling a testing story. Therefore, anyone who declines to explain himself when asked about his testing is not practicing RST. I disavow such testers. (However, just because explaining oneself is an important part of testing doesn’t mean a manager can insist on arbitrarily voluminous documentation or arbitrary metrics. I suspect that, in some cases, managers who complain about testers refusing to document or explain themselves are really just obsessed with a specific method of documentation and refusing to accept other viable solutions to the same problem.)
  5. In RST we say that testing cannot be automated, and that tools can become an obsession. This leads some to think I am against tools. No, I am against bad work. Unfortunately, some tools, such as expensive HP/Mercury tools, are often used to wastefully automate weak fact checking at he expense of good testing. Yes., tools and the technical skills to create and apply them play an important role in great testing. It’s not automating testing when I use tools, because testing is whatever testers do, not what tools do. Therefore a tester who refuses to learn and use tools in general is not practicing RST.
  6. In RST we distinguish between checking and testing. This allows us to distinguish between a test process that is appropriately thoughtful and deep, and one (based solely on checking) that would be reckless and shallow. But when we criticize a checking-only test strategy, some people get confused and think we are criticizing the presence of checking rather than the lack of testing. Therefore, a tester who refuses to design or perform checks that are actually economical and helpful is not doing RST.
  7. In RST, we ban unscientific, abusive attempts at using metrics to control the test process. But when some people hear us attack, say, the counting of test cases, they assume that means we don’t believe in even the concept or principle of measurement. Instead, we support using inquiry-focused metrics (which inspire questions rather than dictating decisions), we promote active skepticism about numbers applied to social systems, and we promote the development of observation, reasoning, and social skills that limit the need for quantification. Therefore any tester who simply refuses to consider using metrics of any kind is not doing RST.
  8. Some people hear about the freedom of exploratory testing, and they confuse that with irresponsibility. But that’s silly. If you drive a car, you are free to run over pedestrians or smash into buildings– except you don’t, because you are responsible! Also, it’s against the law. Freedom is not the same thing as having a right. Therefore, anyone who accepts the freedom of exploratory testing and cannot or will not manage that testing appropriately is an incompetent or irresponsible tester.

A Satisfying News Story

Here is the difference between happiness and satisfaction:

Consider this quote is from How Zynga went from social gaming powerhouse to has-been.

“During my five-month mark, it started turning sour when we were pushing a lot of code that was destroying the ecosystem—they were not fixing bugs,” he said. “At one time, I had 10,000 players trapped inside a quest. 10,000! The attitude was ‘Don’t worry about them.’ [Management] would rather grab new players, keep them for three months or so, get $5 to $10 from them, and those players would quit and leave.”

When I read this, I am not happy. It’s sad to see hundreds of jobs lost because– among other things– a company loses interest in producing a product that works well.

But I am satisfied when yet another example of a high flying wipeout shows that testing (and bug fixing) matters.

Seven Kinds of Testers

Most of my work is teaching, coaching, and evaluating testers. But as a humanist, I want to apply the Diversity Heuristic: our differences can make us a stronger team. That means I can’t pick one comfortable kind of tester and grade people against that template. On the other hand, I do see interesting patterns of skill and temperament among testers, and it seems reasonable to talk about those patterns in a broad sense. Even though snowflakes are unique, it’s also true that snowflakes are all alike.

So, I propose that there are at least seven different types of testers: administrative tester, technical tester, analytical tester, social tester, empathic tester, user, and developer. As I explain each type, I want you to understand this:

These types are patterns, not prisons. They are clusters of heuristics; or in some cases, roles. Your style or situation may fit more than one of these patterns.

  • Administrative Tester. The administrative tester wants to move things along. Do the task, clear the obstacles, get to “done.” High level administrative testers want to be in the meetings, track the agreements, get the resources, update the dashboards. They are coordinators; managers.  Low level administrative testers often enjoy the paperwork aspect of testing: checking off boxes on spreadsheets, etc. (I was a test manager for years and did a lot of administrative work.) Oh, and by “high level” I don’t necessarily mean “high ranking.” I mean someone who prefers to deal with the big picture of testing, but not so much the details of each test. “Low level” means focusing on each click and keystroke; each testing moment. Warning: Administrative testers can be tempted to “fake” the test process– to move things along at the expense of the quality of the work.
  • Technical Tester. The technical tester builds tools, uses tools, and in general thinks in terms of code. They are great as advocates for testability because they speak the language of developers. The people called SDETs are technical testers. Google and Microsoft love technical testers. (As a programmer I have one foot in this pattern at all times.) Warning: Technical testers are often tempted not to test things that can’t easily be tested with the tools they have. And they often don’t study testing, as such, preferring to learn more about tools.
  • Analytical Tester. The analytical tester loves models and typically enjoys mathematics (although not necessarily). Analytical testers create diagrams, matrices, and outlines. They read long specs. They gravitate to combination testing. (If I had to choose one category to be, I would have to say I am more analytical than anything else.) Warning: Analytical testers are prone to planning paralysis. They often dream of optimal test sets instead of good enough. If they can’t easily model it, they may ignore it.
  • Social Tester. The social tester wants you! Social testers discover all the people who can help them and prefer working in teams to being alone. Social testers understand that other people often have already done the work that needs to be done, and that no one person needs to have the whole solution. A social tester knows that you don’t have to be a coder to test– but it sure helps to know one. A good social tester cultivates social capital: credibility and services to offer others. (I follow a lot of the social tester pattern. My brother, Jon, is the classic social tester.) Warning: Social testers can get lazy and seem like they are mooching off of other people’s hard work. Also, they can socialize too much, at the expense of the work.
  • Empathic Tester. Empathic testers immerse themselves in the product. Their primary method is to empathize with the users. This is not quite the same as being a user expert, since there’s an important difference between being a tester who advocates for users and a user who happens to test. This is so different from my style that I have not recognized, nor respected, this pattern until recently. People with a non-technical background often adopt this pattern, and sometimes also the administrative or social tester pattern, too. Warning: Empathic testers typically have a difficult time putting into words what they do and how they do it.
  • User Expert. Notice I did not say “user tester.” User experts may be called domain experts or subject matter experts. They do not see themselves as testers, but as potential users who are helping out in a testing role. An expert tester can make tremendous use of user experts. Warning: User experts, not having a tester identity, tend not to study or develop deep testing skills.
  • Developer. Developers often test. They are ideally situated for unit testing, and they create testability in the products they design. A technical tester can benefit by spending time as a developer, and when a developer comes into testing, he is usually a technical tester. Warning: Developers, not having a tester identity, tend not to study or develop deep testing skills.

When I’m sizing up a tester during coaching. I find it useful to think in terms of these categories, so that I can more efficiently guess his strengths and weaknesses and be of service.

Do you think I have missed a category? Do you think I have de-composed them poorly? Make your case in the comments.

 

Programmer Pairing with a Tester

My sister, Erica, is not a programmer. Normally she’s not a tester, either. But recently she paired with me, playing a tester role, and spotted bugs while I wrote in Perl. In the process, it became clear to me that testers do not need to become programmers in order to help programmers write programs in real-time.

The Context

While working on the report for the Rapid Testing Intensive, recently, I needed a usable archive of the materials. That meant taking all of the pages, comments, and attachments out of my Confluence site and putting them in a form easier to shuffle, subdivide, organize, refer to, and re-distribute. It would be great if that were a feature of Confluence, but the closest I can get to that is either manually downloading each item or downloading an entire archive and dealing with a big abstract blob of XML and cryptically named files with no extensions.

(Note to Atlassian: Please enhance Confluence to include a archivist-friendly (as opposed to system administrator-friendly) archive function that separates pages, attachments, and comments into discrete viewable units with reasonable names.)

The Deflection

While Erica catalogued the names of all the attachments representing student work and the person or persons who created them, I was supposed to write a program to extract the corresponding material from the archive. Instead, I procrastinated. I think I checked email, but I admit it’s possible I was playing Ghost Recon or watching episode 13 of Arang and the Magistrate on Hulu. So, when she was ready with the spreadsheet, I hadn’t even started my program.

To cover my laziness, I thought I’d invite her to sit with me while I wrote it… you know, as if I had been waiting for her on purpose to show her the power of code or whatever. I expected her to decline, since like many computer power users, she has no interest in programming, and no knowledge of it.

The Surprising Outcome

She did not decline. She sat with me and we wrote the program together. She found six or seven important bugs while I typed, and many other little ones. The programming was more fun and social for me. I was more energized and focused. We followed up by writing a second, bigger program together. She told me she wants to do more of this kind of work. We both want to do more.

A Question

How does someone who knows nothing about Perl programming, and isn’t even a tester, productively find bugs almost immediately by looking at Perl code?

That’s kind of a misleading question, because that’s not what really happened. She didn’t just look at my code. She looked at my code in the context of me talking to her about what I was trying to do as I was trying to do it. The process unfolded bit by bit, and she followed the logic as it evolved. It doesn’t take any specific personal quality on the part of the “coding companion,” just general ones like alertness, curiosity, and basic symbolic intelligence. It doesn’t take any particular knowledge, although it can help a lot.

Perhaps this would not work well for all kinds of coding. We weren’t working on something that required heaps of fiddly design, or hours of doodling in a notebook to conceive of some obscure algorithm.

My Claim

A completely non-technical but eager and curious companion can help me write code in real-time by virtue of three things:

  1. The dynamic and interactive legibility of the coding process. I narrate what I’m doing as it comes together step-by-step. The companion doesn’t eat the whole elephant in a bite; and the companion encounters the software mediated by my continuous process of interpretation. I tell him what and why and how. I do this repeatedly, and answer his questions along the way. This makes the process accessible (or in the parlance I like to use “legible” because that word specifically means the accessibility of information). The legibility is not that of a static piece of code, sitting there, but rather a property of something that is changing within a social environment. It’s the same experience as watching a teacher fill a whiteboard with equations. If you came at the end of the class, it would look bewildering, but if you watched it in process, it looks sensible.
  2. The conceptual simplicity of many bugs. Some bugs are truly devious and subtle, but many have a simple essence or an easily recognized form. As I fix my own bugs and narrate that process, my coding companion begins to pick up on regularities and consistency relationships that must be preserved. The companion programs himself to find bugs, as I go.
  3. The sensemaking faculties of a programmer seeking to resolve the confusion of a companion. When my dogs bark, I want to know why they are barking. I don’t know if there’s a good reason or a bad reason, but I want to resolve the mystery. In the course of doing that, I may learn something important (like “the UPS guy is here”). Similarly, when my coding companion says “I don’t understand why you put the dollar sign there and there, but not over there” my mind is directed to that situation and I need to make sense of it. It may be a bug or not a bug, but that process helps me be clear about what I’m doing, no matter what.

And Therefore…

A tester of any kind can contribute early in a development process, and become better able to test, by pairing with a programmer regardless of his own ability to code.

Rapid Software Testing at Barclays

I’m excited to be working with Barclays on an unprecedented project: creating a professional testing culture based on the Context-Driven principles and my Rapid Software Testing (RST) methodology. The Barclays Global Test Centre (GTC), led by Keith Klain, has hundreds of testers spread around the world. They work in a regulated industry on high stakes products. But unlike nearly every other large organization in the world, they have decided not to rely on pretense and 40 year-old ideas that were discredited 30 years ago. They are instead putting in place a system to recruit and grow highly skilled and highly motivated testers.

Barclays’ approach in the GTC is to identify and encourage dozens of testing champions in its ranks who are the role models and mentors for the rest of the group. Anyone may aspire to be in this special group, but to be recognized requires that the candidate tester demonstrate vigorous self-education and critical analysis. Some of the testers in the group began as strong skeptics of Rapid Testing. But the methodology is designed for skeptics– it is based on skill development and heuristics rather than pushing “best practices.” In Rapid Testing, the skilled tester is always in charge, not pieces of paper or officious charts.

RST requires each tester to employ his own judgment and technical analysis, much like what airlines expect of pilots, or hospitals expect of doctors. That can’t work on a large scale without a strong corporate commitment to training and personal ethics. Management must drive out fear, so that testers are willing to take the sort of risks that come from making their own decisions about test strategy. But the onus is on the testers to earn personal credibility within an internal community that can effectively police itself. Any tester, at any time, is expected to stand up and explain and defend his work.

I’m aware of only two large companies in the world that have made a commitment to this kind of professionalism, which is an altogether different sort of professionalism than the ceremonial certification variety that is promoted by most organizations. In Barclays’ case, this commitment has strong support from top management, and I have personally witnessed, in my weeks of working with them, that the testers at their Singapore operation have fire in their eyes. There are testers here who deserve to have an international reputation.

Thoughts Toward The Ethics of Testing

I am thinking and talking a lot about ethics, lately. Maybe that’s because Context-Driven testing is getting better traction, now. More testers are approaching me, asking for guidance. More testers are challenging fake testing in their organizations.

Or maybe I’m just noticing it more, because Cem Kaner used to take the brunt of all this. In that respect, I have historically been more a follower than a leader in our community. Recently, I’ve stepped forward to be more of the role model that I am expected by my colleagues to be.

(Note: Ethics is a guaranteed sore subject. Whenever I talk about ethics, I can’t avoid implying that I believe some people are not as ethical as they should be. But, you know, that’s the burden of leadership. To those who avoid politically difficult subjects, enjoy your shadowy existence. Lurk on, yon lurkers.)

I have an ethical code. Actually, several! The Association for Software Testing adopted the ACM Code of Ethics, verbatim. It’s okay. I would have preferred the simpler IEEE Code, personally. I also almost completely follow Jerry Weinberg’s code. I take comfort from all of them, plus I have my own.

I need my own, because these codes above don’t directly deal with certain common testing ethics issues.

So, I’m the content-owner for the 2nd Kiwi Workshop on Software Testing, tomorrow morning, here in Wellington, New Zealand. Our topic is ethics. To get ready for it, I thought I would write out some of the ethical principles by which I strive to operate. Here they are:

  • Know what a test is. Avoid labeling an activity as a “test” unless it represents a sincere effort to discover a problem in a product.
  • Maintain a reasonable impartiality. The purpose of testing is to cast light on the status of the product and its context, in the service of my clients. I may play multiple roles on a project, but my purpose, insofar as I am a tester,  is not to design or improve the product.
  • Do not claim to assure, ensure, or control quality. I don’t control anything about the product: a tester is a witness. In that capacity, I strive to assist the quality creation process.
  • Report everything that I believe, in good faith, to be a threat to the product or to the user thereof, according to my understanding of the best interests of my client and the public good.
  • Apply test methods that are appropriate to the level of risk in the product and the context of the project.
  • Alert my clients to anything that may impair my ability to test.
  • Recuse myself from any project if I feel unable to give reasonable and workman-like effort.
  • Make my clients aware, with alacrity, of any mistake I have made which may require expensive or disruptive correction.
  • Do not accede to requests by my client to work in a wasteful, dangerous, or deceptive way. (e.g. I will not keep test case metrics, because they are damaging in almost any context)
  • If I do not understand or accept my mission, it shall be my urgent priority to discover it or renegotiate it.
  • Do not deceive my clients about my work, nor help others to perpetrate deception.
  • Do not accept tasks for which I am not reasonably prepared or possess sufficient competence to perform, unless I am under the direction and supervision of someone who can guide me.
  • Study my craft. Be alert to better solutions and better ways of working.

 

 

 

Bharani Asks About Daily Practice

Bharani from India is a novice tester who appears to have a lot of energy. She recently emailed me this:

Sir, I need one small tips. As a, new tester how should they improve to became sharp and clever in the testing field.
 I mean that, What is their homework. what they should do daily…without skipping
For ex: like normal daily activities. we have to do exercise  breakfast..so….so..
I mean that, what is the testers daily activities?

I am asked many questions by testers. This particular one reveals a lot about the questioner. For one thing, she thinks differently, because I have never been asked this. And it’s a question about discipline, so perhaps she’s a tester who is truly serious about developing herself.

My first answer is the same as the one my world famous writer of a father so often gives: write.

Write every day.

For me this takes the form of carrying my Moleskine notebook everywhere I go (now with a bandolier for my pens!). Whenever I find myself with a few moments, I make notes of my thoughts about testing and technical life. My notebooks serve as one source of great new ideas for my consulting and teaching.

But for testers, there are a few more practices to keep in mind…

Watch yourself think ever day.

While you are working, notice how you think. Notice where your ideas come from. Try to trace your thoughts. This is not easy. You have to practice to improve your skill of self-observation.

As a tester, you must become an artist of your thoughts, you must learn to see the structure and gain control of the structures by which you notice what no one else notices. These structures are not simply “intuition” or “common sense.” We can do better than that.

Question something about how you work every day.

Testers question things, of course. That’s what testing is. But too few testers questions how they work. Too few testers question why testing is the way it is.

For example: Why do we talk about tests “passing” and “failing”, instead of talking about whether there are problems? Do all bugs need to have steps to reproduce? How do we decide when to write a test down and what steps need to be written?

Explain testing every day.

The ability to explain testing is linked to nearly every other testing skill. Even if no one makes you explain your methodology, you can explain it to yourself. Do that. Practice doing it by voice and in writing. Don’t rest with buzzwords. Don’t tell yourself that you did “black box testing.” Break it down. Explain what you did specifically and explain why you did it.

 

Mechanical or Magical? Noah Says “Neither.”

As I was having dinner with Noah Höjeberg tonight, he said an interesting thing. “Some people think testing is mechanical, and that’s bad enough. But a lot of people seem to think the alternative to mechanical is magical.”

(Noah is the new test strategist at Scila AB, in Stockholm. Interesting guy. I’ve played a lot of testing dice with him, in the past. I meant to do the Art Show game with him, too, but we got so much into our conversation that I completely forgot.)

Mechanical and magical are false opposites. In Rapid Testing, we pursue another path: Heuristical. In other words, skilled testing, achieved through systematic study and the deliberate application of heuristics. This is neither a mechanical, algorithmic process, not is it magical, mystical. We can show it, talk about it, etc. And yet it cannot be automated.

How I Invented Sympathetic Testing

I did not invent sympathetic testing. Anyone who says I claim to have invented it will have only read the title of this post, but nothing further. Now you know.

I may have been the first in my circle to recognize one specific benefit of sympathetic testing. But if so, that is a minor technical point. Because I know that my recognition came directly from a point of view championed by Cem Kaner. From where I stand, it came largely from him.

Okay, James, this is a bit confusing. Why are you talking about this, today?

Ajay Balamurugadas came to me today asking if I came up with the idea of “sympathetic testing.” This is an idea he’s found in Mike Kelly’s work, which derives partly from my work (on tours). It’s also in the section of Lessons Learned in Software Testing that I wrote (and which was edited by Cem Kaner and Bret Pettichord). He had already asked Cem Kaner, who said he got it from my brother [correction: Ajay tells me he saw Cem say that on a video]. But if so, I know exactly where Jon got it: from me, during a project we worked on around 1997. That’s where he was working for me on a court case where I had to do an analysis of “good enough” quality, using a method strongly influenced by Cem’s thinking about cognitive biases.

I’m talking about this today because the provenance of ideas can be confusing when people work closely together. Who cares who invented it, or named it? You might be surprised. Credit is important to people who live by their reputations in a world of ideas. Reputation is money, to put it bluntly. To achieve high income and control over your work as a tester, one way– and the only way I know, actually– is to build your public reputation. Moreover, a lot of our motivation is the respect of our peers. Therefore, this matters.

This isn’t really about “sympathetic testing”, then. Still, what IS sympathetic testing?

Sympathetic testing, as I think of it, is sometimes slightly confused with the closely related ideas of “happy path” or “positive testing.” Sympathetic testing means testing while affirming (rather than challenging) each assumption, resource, or service. But if that’s all it meant, then it would be the same as positive testing. For me, sympathetic testing means more. It means asking “what is wonderful about this product?” rather than “what is broken about this product?” I can do positive testing all day and still be focused on bugs. With sympathetic testing, I may find bugs, but that is not my focus or purpose– my main purpose is learning; building a rich model in my mind.

The insight I had, when working with my brother (way back before he was a recognized test expert in his own right) was that sympathetic testing made me better at unsympathetic testing. Searching happily prepares me to search aggressively.

Okay, then what’s really troubling you?

What bugs me is that Cem deserves more credit for sympathetic testing. But if he claims it himself, he probably thinks he will anger me, and if he gives me full credit, it would anger him: because Cem and I aren’t on speaking terms, at the moment. (I hope that’s temporary.)

I’m hereby offering him credit. Cem and I collaborated for roughly 16 years. We had probably hundreds of deep conversations about testing. Among those conversations, he lectured me on mental models and biases. We once had a fairly bitter argument about what a model is. He won that argument, thankfully, and I have been comfortable with the outcome ever since. He introduced me to Cognitive Science and pushed me deeper into Epistemology. It was that sensibility that led to me discovering a lot of things: among them the power of sympathetic testing. I’m certain that the first person I ran to with my discovery was Cem, but not only that, this is exactly the sort of thing that he was famous for encouraging: to look at things in a sympathetic way. I can’t remember the conversation itself or what he said. I bet he said something like “That’s just what I’ve been trying to tell you!”

Therefore, “sympathetic testing” is part of our joint work. Between about 1997 and 2007, it’s a good bet that I didn’t have a significant thought about testing which wasn’t also sifted and tested by Cem. We didn’t always agree, but we were extremely close. It’s for that reason that he put my name on his BBST class, even though I didn’t do any direct work on it. He was recognizing my influence on him. I’m doing that now.

I’m my own man. I have reinvented testing for myself (just as I recommend that others do). And yet a few people have deeply influenced me in that process. The three people who have had the most influence on my are my father, Jerry Weinberg, and Cem Kaner. I sometimes joke that I can sell out and offer baloney certifications to gullible testers, just like the ISTQB does– but only after those three men have died and I’m no longer seeking their respect.

[Postscript: Why did I title this piece “How I Invented Sympathetic Testing”? Because it was the best way I could think of to attract the attention of people who might be upset that I would make such a claim. Then maybe they would read this post, and feel a lot better.]

What We Read

I staggered out of the Cambridge Press bookstore a bit dazed, today, having gorged on 21 books. [Addendum: I mean by this that I browsed them, purchased them, and had them shipped home.] If you want to know what a Context-Driven tester reads, here it is:

  • A First Course in Statistical Programming with R
  • Wisdom: Its Nature, Origins, and Development
  • Learning and Expanding with Activity Theory
  • Sequential Analysis and Observational Methods for the Behavioral Sciences
  • Observing Interaction: An Introduction to Sequential Analysis
  • Human Error
  • Combinatorics: A Problem Oriented Approach (Mathematical Association of America Textbooks)
  • A Mathematician Comes of Age (Spectrum)
  • The Golem at Large: What You Should Know about Technology (Canto)
  • The Golem: What You Should Know about Science (Canto)
  • A Practical Introduction to Denotational Semantics (Cambridge Computer Science Texts)
  • Dialogic Inquiry: Towards a Socio-cultural Practice and Theory of Education (Learning in Doing: Social, Cognitive and Computational Perspectives)
  • From Teams to Knots: Activity-Theoretical Studies of Collaboration and Learning at Work (Learning in Doing: Social, Cognitive and Computational Perspectives)
  • Nuts and bolts for the social sciences
  • How to Fold It: The Mathematics of Linkages, Origami and Polyhedra
  • The Cambridge Handbook of Thinking and Reasoning (Cambridge Handbooks in Psychology)
  • The Manuscript of Great Expectations: From the Townshend Collection, Wisbech (Cambridge Library Collection – Literary Studies)
  • The Cognitive Basis of Science
  • Satisficing and Maximizing: Moral Theorists on Practical Reason
  • The Cambridge Handbook of Situated Cognition (Cambridge Handbooks in Psychology)
  • Risk Communication: A Handbook for Communicating Environmental, Safety, and Health Risks

One of the challenges I have for the ISTQB proponents is “What do you read?” You see it’s a trap. If they tell me they read widely, deeply and liberally, I contrast that with the intellectual desert that is the ISTQB Syllabus and ask them why there is such a disconnect between their education and their professional claims. And if they read narrowly, well, there you go.

If you want to be an excellent tester, you need a good education. You didn’t get that in school (or if you’re in school, you’re not getting it), so you need to do something like what I do: scout for fabulous and offbeat books about all the matters of great testing– and testing touches EVERYTHING!

[addendum: If you are not familiar with my distaste for institutional education, before picking a fight with me, go see my book, Secrets of a Buccaneer-Scholar. I spent 26 years doing the research by which I assert that school, although not always destructive and occasionally helpful, is certainly not necessary if you want to live a successful intellectual life. Each day of my life is another data point about how wrong were the teachers who told me I would not be successful without submitting to “the game” of school they desired me to play.]

Most of the books on my list are self-explanatory. One in particular may seem strange: the manuscript of Great Expectations. I picked that one up because the photographic images of Dickens’ original manuscript is a beautiful example of how messy the creative process is. Imagine trying to put metrics on the process of writing that, with all its crossouts and insertions.Writing is exploratory. Just. Like. Testing.