I am privileged to be mentioned in one of your blog posts! Thank you very much. I will not disappoint you and I promise to keep blogging

Regards,
Nandagopal

I have to agree with You in essence that none certification could ever replace experience on any field. In the other hand, You have to understand that none certification, even University diploma, never ever was intended to replace exp. In such, certification is only reference point to knowledge required by particular program, ie.: ISTQB and should be treated only this way.

[James' Reply: I think you missed my point. I'm not complaining about the limitations of the ISTQB, but about the stupidity and injustice of it. How would you like it if I hit you in the face, and then when you complained I were to say "I never claimed that hitting you in the face was a perfect solution to my problem, nor did I claim that it would be pleasant for you. The difficulty is in you, because you don't like being hit in the face with my fist."]

Difficulty is in people. They are week beings that has to believe in something. Some believes in God others in money and anothers in politicians. So the real thorn is not in certification programs, ok they could be better, but in the way like people treated them. We can ban certifications at all but do we realy have enough strong to change milions of people?

[James' Reply: Let's begin by banning bad certification programs and worry about changing people, later.]

I want to answer on Harinder Seera question: “How would you test something that you can not test?”

Answer is very simple - You have to place object in some well defined environment and observe this env. I do not know the English equivalent but in Polish we say “Metody Po?rednie”.

As you say, not only it is hard for one routine to know what will be the next routine called, but doing that would wreck modularity cohesion and low-coupling :).

But for most part (perceiving delays, or analyzing hangs for example) logging at the beginning of each routine would be enough.

[James reply: Okay, so logging "I'm about to try this... if you don't hear from me, tell my wife I love her..."]

Looking forward to learn and contribute more and better towards building a bigger Sapient Testing community including Passionate Testers from India.

Thanks once again for your time and effort you put in build a sapient testing community.

NOTE : The CAPTCHA I filled for this comment is to daydream,but my dream came true today

Regards,
Dhanasekar S
Another Passionate Tester of India
http://testingideas.wordpress.com

If I may, let me add a 14th point, an insight I learned from a tester in my team:
For functions the software does by itself, in addition to logging the result of each step after it happens (imported data; generated report; exported graph…) it is very useful to log *what the next step will be*. This way, when a failure occurs, it is easy to know what the software was trying to do.

Since I learned that, I am paying attention to it, and impressed by how much common logs or progress reports lack this.
It takes a lot of time for an operation to happen, but you need to wait until the end of it to know which it was.
Once my computer froze during Windows’ shutdown with the “playing logoff sound” screen. I’m pretty sure the sound playing was over, and was left wondering what caused the OS freeze.

Hope you approve the addition

[James' Reply: Let's say I'm a programmer. How would I actually implement this? Are you just saying you want me to log the start of events, instead of just logging when things end?

Obviously, I can't know, from within a specific routine, what the next thing will be.]

I was wondering, if you would be able to provide me some examples on how best to isolate defects between more than two components. (Assuming that such defects in some way maybe considered as intermittent)

Regards
Gaurav Pandey

[James' Reply: Can you present that as a more natural problem? What would it look like in real life?

If you see something that is intermittent, you won't know which variables make it intermittent, or how many they are.]

On the other hand, though, (and I know nothing about planes) would you then have done the manual checks too? Or only done manual checks if the automated tests had flagged something. I try for the former, but am probably guilty of the latter too often for comfort.

A very good and timely reminder to keep alert!

[James' Reply: A sapient pilot is always needed. Automation is fine as long as you don't mind when it fails. We have lots of automation on our cars, for instance, but failure there generally leads to being stuck at the side of the road. Automation failure on aircraft is considerably more risky. That's why we pilots are constantly watching and checking and cross-checking.]

James, I really liked your statement above. Since years you seem like the one of the few serious voices advocating automation WHEN it makes sense - and not just everywhere it might. The ROI in automation is indeed an issue and should be taken more seriously. Everybody can sell automation - it sounds so nice specially to those not in the know - and the dream of the magic button.

Funny how even after James gave, in his story, explicit critical criteria that pointed to the problem—color, taste, smell—they still don’t appear on the list.

The argument is simple: unlike humans, automation doesn’t have expectations. It is capable of detecting matches or mismatches based on the expectations we feed it. Humans, by contrast, can identify expectations prospectively, in advance of an observation (”If this were to come to pass, it would indicate a problem”); on the fly, during an observation (”Hey… I didn’t expect that; that could be a problem”); and retrospectively (”Now that I’ve seen the problem, I can figure out how I might have detected it” or “Now that I have a better model, I realize that I could have tested for this, even though a problem hasn’t manifested itself yet.”).

Just light the red lamp for a pilot if a parameter changed to a value outside of defined range.

One indicator of a problem with any analysis is the appearance of the word “just”. Yes: lighting a red lamp is a simple enough thing. Knowing when to light it and when not to light it is the complicated bit. Interpreting it is also complicated. Here’s an example of a one-bit communications protocol: someone honks the horn at you, as you’re driving. How would one interpret accurately that without a walloping dose of context to go along with it?

—Michael B.

[James' Reply: I would have to say no and yes. I went through some personal upheavals that knocked my mindset around for a while. However, I did not gain back all the weight, and as of last month, my brain clicked into the groove again and I am back to walking each day. I had a special tray built for my treadmill, and it holds my computer. I've discovered that I can put set the treadmill to one mile per hour and have no problem working while walking. As I'm typing this, I'm doing 1.3 MPH.]

This is the lesson to think about…

“Crash pilots given conflicting orders”
http://news.bbc.co.uk/2/hi/europe/2115425.stm

[quote]
German investigators have revealed that the pilots of the Russian airliner involved in last week’s mid-air collision with a cargo plane received contradictory instructions seconds before the crash.
They said voice recorders recovered from both aircraft showed Swiss air traffic controllers told the Russian pilots to descend, while the on-board warning system instructed them to climb.

All 69 people - including 45 schoolchildren - aboard the Russian Tu-154, and two crew members on the Boeing 757 were killed when the two aircraft collided at 35,000 feet (10,500 metres) over the German-Swiss border.

According to German authorities, cockpit warning systems told the Tu-154 to climb and the cargo jet to descend, just 45 seconds before the collision. But voice recorders reveal that one second later, Zurich air traffic controllers told the Russian pilots to descend. The Russian crew did not respond, so the Zurich control tower repeated the order 14 seconds later, investigators say.

The Russian plane responded and the two aircraft collided 30 seconds later.

Although the aircraft were flying over Germany at the time, they were under the control of the Swiss air traffic control body, Skyguide.

[/quote]

Thank you,
Albert Gareev

[James' Reply: It is customary to call an example like this a lesson. But really, it's not a lesson. This is a story. It becomes a lesson when we interpret it and make it a part of ourselves. My interpretation of this, I suspect, is different from your interpretation.

I see this as a lesson about the automation bias and the need for vigorous training in order to pilot aircraft safely.]

Great story today. Your one-liners such as:
“Unless you landed at the rainbow lake on Unicorn Planet, there may be a hydraulic leak in there.” or
\In that way, we are like secret service agents, trained to deal with known threats and to have a good chance to discern fresh and unusual threats, too.\
makes me smile all day in office.
Back to testing
Cheers

“The question “Can I explain everything that I see?” is a good heuristic to keep in mind while testing.”
I find this heuristic to be valuable. In the past, I’ve seen an anomaly, say, “yesterday the program wasn’t sending me emails, but today it is.” It’s insanely easy to say “Well, it’s working now, so it passes, and that’s ok.” I always try and make a point now of asking “why was it not working yesterday, but today it is?” rather than just being content that it now works. The investigative path often leads to very interesting information, even if you never end up explaining the anomaly.

Hi James,

You suddenly decided to pump the float (the action wasn’t really required) and discovered an issue that could have become a serious problem. That is a typical situation in testing but I don’t see why you [hypothetically] have problems imagining automation of float pump tests that would find unexpected/undefined problem. Below is the simplified illustration.

Layer One: Detection
I’m sure you’ve heard of Boolean algebra and I’m 100% confident you practically used elements of it as you did some programming.

[James' Reply: This is how you're going to start your argument? Okay. Why yes, Albert, I have heard of Boolean algebra. It came in handy when I was writing machine code for those video games I wrote for the Apple II, Commodore64, PC, and Amiga, back in the eighties. And indeed it's essential in writing the test tools and automation that I create, today. I sure hope the point you are about to make is better informed about computer science than it is about my technical background...]

The main rule that you would need to use here is “FALSE = NOT TRUE”
Implementation would require heuristic too: instead of looking for problems automatic unit must look after the object’s (float, pump, and peripherals) state. If object’s state is deviated from the base one - that is an indicator of [possible] problem. State should be defined by a sufficient set of criteria: amount of water in a float, chemical activity of water (pH, electro-conductivity, etc), vibration, pump performance, - as much as required.

[James' Reply: This fails in two ways to address the issue I raised, both of which you need to understand if you want to get good at test automation:

There are certainly situations we both could cite where we would think that any change to a particular variable is highly likely to be bad. However, in my experience working with complex dynamic systems and complex testing thereof, those situations are not commonplace. Frequently it requires clever coding to mince our way around things that are fully expected to change. The more you write such code, the more expensive your automation becomes.]

//Note. That already could be sufficient enough, depending on context. Just light the red lamp for a pilot if a parameter changed to a value outside of defined range. However, we may end up with too many lamps or red lights indicating low threat that could have been ignored if checked by a human being. To avoid that problem an automatic checking unit should become a little smarter on its own and do evaluation.//

[James' Reply: In the cockpit, I have a number of instruments that can indicate potential trouble, but just that indication-- which is all automation can do-- is not enough for excellent flying (or testing). There is no "everything is okay" detector on the plane. Nor can there be. The designers have anticipated various specific kinds of problems and I have detectors for those (a stall warning tone, for instance). But on top of all that-- and here is the point I'm trying to make-- my sapience must interpolate and extrapolate; generate ideas and filter them. The automation can help, but it cannot replace sapience.]

Layer Two: Evaluation
Boolean algebra is not sufficient enough here. Fuzzy Logic methods (or Artificial Neural Network for more complicated cases) are needed to be used.

[James' Reply: I would argue that a natural neural network is needed.]

From the context of your story: small amount of water in a float isn’t a problem itself; a little salty water is also a valid case. That is, “true” or “false” state of event doesn’t give basis for evaluation. It’s combination of events and weight of each event that matter.
Examples.
Little water + high electro-conductivity = alert
Little water + high/low pH = alert

[James' Reply: What is the false positive rate for that and wouldn't it be easier just to taste the water? This also demonstrates my point about pink water: I don't see anything here that detects pinkness. Here's some homework for you. At minute 27 of this video, this doctor explains how a particular form of automation has made medical testing WORSE.]

//Note. It should be seen pretty clear now that implementation of such a solution isn’t a quick patch. Therefore it should be reasonable and affordable.//

From the safety and security stand points it could be reasonable even for a light aircraft. Economically it could be affordable if it’s sold along with 100 of planes. Or 100,000. Or one large plane.

[James' Reply: But Albert, your very example suggests that it's intractable, because you are either unwilling or unable to supply me with a complete list of things to check for and a complete design of the equipment to check for it. Oh, and how much will this equipment weigh? How expensive is it to test and maintain? Simplicity and lightness is important in designing light aircraft.

You haven't addressed these concerns, nor would I expect you to, because my argument is that it's ridiculously hard to do so, and not at all worth doing compared to training the pilot to handle these things himself.]

It also may lead to what I wouldn’t want to happen but what could happen and happens anyway.
For example - replacing a highly qualified mechanic with a combination “automatic unit” + “2-week trainee”. It could be because it’s too hard to get or too long to grow a professional, or because it’s much cheaper to employ 10 surrogates. Or in an organization with extremely high turnover of staff. That used to be only army in action but now seems to be any big corporation.
This approach has come and stayed, and it looks very attractive in accounting books. What is not visible in balance sheets that professional armed with automation can bring performance to new levels of excellence while amateurish workers even with automation have problems performing just as good.

So my point is that instead of trying to stop the progress maybe better to walk ahead of it?

[James' Reply: What do you mean progress? I'm not standing in the way of any progress. But I would say that it is not progress to seek to do with automation expensively and poorly what can be done far better and far less expensively by a skilled tester.]

(Remember, Mars Pathfinder over-succeeded in 1997, working automatically, without direct radio guidance)

[James' Reply: Yes, when I taught my testing class at JPL and saw the new rover under construction. That's wonderful and extremely expensive technology. It also had a critical fault in it that almost ended the mission on the second day!

Dude, I'm not opposed to test automation. I'm opposed to wild irresponsible claims made about test automation. This is based on extensive experience with test automation that I have, as a programmer and as a tester and as a consultant who has written reports about test automation failures.]

Thank you,
Albert Gareev
http://automation-beyond.com/