Could you please explain in brief (one or two lines) what is ‘inherently testable’.

Thanks.

[James’ Reply: “Inherently” is a bit redundant. I could have just said “testable”. What I mean, simply, is that the product is designed with testing in mind. This primarily means controllability and visibility features. For example: log files (visibility of internal states) and a scriptable interface (controllability).]

[James’ Reply: I actually don’t have a list of favorite sites! I just go to Google when I want something. I have to think more about this.] 

[James’ Reply: I would need you to explain what you mean by traditional testing and creative testing. I would also need you to describe what you mean by personality.

In my view, the tradition in testing to is treat the process as a form of secretarial work. I see testing as a matter of systematic skepticism and critical thinking. The personality of a consistently successful tester of this kind must be one of being excited (rather than frightened) by complex problems and ambiguous situations.] 

[James’ Reply: There’s only one context that matters– the one that you are in at the moment. If that changes, then you adjust accordingly. It’s like parenting. You don’t have to figure out how to parent every child, just the ones that belong to you. The context-specific attitude says adapt to your children and then stop adapting. The context-driven attitude, taken to its logical conclusion, is like a child psychologist’s approach. Child psychologists need to know how to adapt to any given child (normal and strange) who walks in the door.

For the same reason, if you figure out how to report coverage on your project in a way that works for you, you can’t assume that the same method will work for me, nor do you need to worry about whether it would work for me. You can’t tell me “James, I have discovered the right way and you should do it my way, too.” What you say, intead, is “James, would you like me to describe some experiences I’ve had with coverage reporting? I feel good about how I do it, over here in my project.”]

I just test ASAP by first preparing a plan and deciding what and how to test.

This time, I tested ASAP without enough planning and time to realize some other things can go wrong.

[James’ Reply: My point is that you can prepare yourself with the knowledge and tools to test much quicker, yet better, than traditional testing claims that you can. In particular, I’m saying that the anxiety and confusion that often come over testers who are under time pressure need not be a problem for you.]

I know I have a lot to learn in order to become a good tester, so this episode will help me for sure.And I agree with you, the lead or manager of the dev is the one responsible, but I would be happy to stop this kind of issues going into production next time.

The lead was aware of the fact I was not the one to be blamed, he came to me and told me next time not to trust what devs write in emails.
I did not have to defend myself, it was a talk trying to improve what goes into production. I did not tell him “next time better include ALL changes in the email”, because he was away when the email was sent. Also, I did not suggest him to have a change management control tool, because at that time he was upset and could have been irritated by such a suggestion. I will talk with him and his team about this in a few days :).

I wrote this here because materials I found here and at Cem Kaner’s site helped me a lot building my confidence. I was doing some things without having “rules” supporting my way of doing things. Then, after reading some articles I found here there were “rules” supporting my way of doing things.

Thank you,
Victor

[James’ Reply: I commend your attitude. Like you, I look at what I can do personally to make things better, without accepting blame for problems other people have created.]

Some time ago, I received an email from a developer which said:

Hi,

Here are the changes I made to the project:
1.
2.
etc.
Please test that ASAP.

The changes were described in detail.

I printed out the email and started verifying each statement. I had some talks with the dev because I did not understood exactly one of the statements.
After I verified it worked correctly when following the statements, I started thinking what could go wrong. I found some issues, related to the changes described in the email.
I continued, and at some point I decided it is good to go.

Today (more than a week later), I found out there was another thing which was modified and broke.
I found this from the dev’s lead.
He told me that even if the dev told me that the only changes were described in the email, he still modified something else. He asked me that next time to test more thoroughly. He also told me not to believe what is written in “change logs”, as some other “minor” changes can be made.

I was mad at me for not catching the issue, as it was an important issue.
I read the email about the changes again and I know why I missed the issue. I misinterpreted it.
The email talked only about the changes the dev “thought” he made. He also did some “minor tweaks” to other parts of the code, but did not mention those in the mail. Also, the email was imperative that the changes have to be tested ASAP. I talked with him at that time and understood there was some kind of pressure from upper levels to get the changes into production.

Now, I did not find a solution to these kind of problems yet, as it has just happened one hour ago.

First, I thought some kind of change management would solve these kind of issues, but then I decided to build rules only for me.
Why?
Because even if someone says: these are the changes, there can be other changes he/she thinks are irrelevant.
Because tiny changes considered “tweaks” will almost always overlooked when writing what was changed.

I just started defining some rules for me so I can minimize the impact of “minor” changes:
1. Don’t test ASAP.
2. If you have to test ASAP buy time.
3. If you have to test ASAP make sure you ask: is this THE ONLY change?
4. Even if someone says IT IS the only change, also test the important parts.
5. If you don’t have time to test the important parts, buy time.

That time I followed rules number 1 and 2, even talked with the developer about the changes, but did not follow rules 3 and 4.
I would have discovered the issue just by hovering the mouse over some part of the product (which I knew is important), but I didn’t.
I concentrated only on the changes described in the email.

I don’t know if the rules I came up will do the job.
Please, if you have other suggestions to improve the way I/we should handle these kind of situations share them with us.

Thank you,

Victor

[James’ Reply: Victor, I’m impressed. I love working with people like you. You coach yourself!

Indeed, I have also found that an off-the-cuff email from a programmer is an unreliable guide to what’s been changed. We can generalize that: all communication from a programmer is unreliable. This is true simply as a consequence of the even more general rule that all human communication is unreliable.

Just remember, you are a tester, not a sucker. I like your rules 3 and 4. Personally, I do test “ASAP”. You can, too. But you need to know how to do it. One thing I do is prepare a test coverage outline and my test data and tools so that I can remember the right things and do the right things under time pressure.

Some people think of test development as creating written test cases. I think of it as preparing for rapidly testing and re-testing my product. This may include automated tests. It definitely includes exploratory testing that adapts to whatever I suspect has changed or been impacted by changes.

It also helps to get information about changes directly from other sources. For GUIs, I have used a tool called Restorator to automatically detect changes in GUI elements by reading the embedded resources. Occasionally I have tracked changes in the source control system as a way to plan testing.

Now the most important thing… While I think you should develop yourself into a tester who can handle anything, the fact is the project manager screwed up. Not the programmer, and not you. If a product is going out very soon after a change is made, then no competent project manager will fail to institute a change control and change review process that is sufficient to assure that each change is approved, reviewed, and tested. (There is an exception to this, of course. If quality doesn’t matter much, then don’t bother with change control.)]

I have following doubts:
1.When should we automate?
2.Once we have the manual test cases, how to check for the feasibility of automating the same/what all should be considered for the feasibility study?

3.And above all, What is the best argument to put to the customer to remove their misconception on automation? Because we really know , the time we spend for developing these scripts , maintain for every release(i.e every 3 months) can be spent usefully on effective and efficient manual testing.

[James’ Reply: I think I’ve covered this subject in my article on agile test automation. You’ll find it at http://www.satisfice.com/articles/agileauto-paper.pdf 

So, what’s your take on creativity in software testing?

Is an environment with humour necessary to foster creative testing?

[James’ Reply: I’m not sure what most people mean by the word creativity. Some people seem to use it to mean the ability to produce new and different ideas. Other people use it to talk about self-expression. For still others it connotes a corrosive fog where intelligence turns to mush.

So, I try to avoid the term. I prefer Ed deBono’s term “lateral thinking” and other descriptive terms such as critical thinking, systems thinking, abductive inference, etc. That said, all of the literature on creative thinking certainly applies to testing. 

I suspect humor helps testing. Look at the nature of humor– a lot of it is the process of making absurd juxtapositions. Just like I do during an input constraint attack.]

“Exploratory Testing is applied Psychology for Testers” - helping to think, analyze and model the problem space. A good ET goes beyond testing as seen from world’s view (external View) and dwells deeply into mind of a testers and attempts to find answers for questions like how good tester test and apply mind in solving testing problems”.

This definition is my answer to a common notion (which you have rightly countered in many places) is that “ET is no big deal - testers (professional !!!?) used this much before the term was invented. ET is a consulting Jargon used by few people to make money”.

What is your opinion on this?

Shrini

[James’ Reply: I would say that testing– all testing, not just ET– is applied Epistemology. It is also, certainly, applied Cognitive Psychology (by definition, all intellectual activities are, since Cognitive Pyschology is the study of how people think).

When people tell me that ET is no big deal because its “already known”, I have to struggle through a couple of reactions before I get to one that is useful. My first reaction is shock, because I’ve been studying and practicing this subject for years. The first useful reaction I generally have to is ask the person to go ahead and explain exploratory testing to me. If they already know it, surely they can teach it to me. Invariably, they are not able to say much about it, and that’s when I can start talking about specific heuristics, terminology, dynamics, and related skills.

Another thing I generally say is that, of course, Cem Kaner and I did not invent the phenomenon of exploratory testing. We invented, and continue to invent, ways of doing it better, ways of talking about it, and ways of teaching it.

As far as money goes, I think anyone who actually looks at this field will immediately see that the people making money on testing are large testing companies, and those companies, as I see it, push expensive methods of heavily documented testing because it’s profitable, suitable for lower skilled workers, and because most of their customers are prepared to believe that scripted testing is good testing, however unjustified that belief happens to be.]